Oct 30 02:38:06.129344 ip-10-0-136-171 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 02:38:06.129354 ip-10-0-136-171 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 02:38:06.129361 ip-10-0-136-171 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 02:38:06.129573 ip-10-0-136-171 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 02:38:16.379027 ip-10-0-136-171 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 02:38:16.379043 ip-10-0-136-171 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9dca3f6045e6495382a3f607a1e996b7 --
Oct 30 02:40:20.581488 ip-10-0-136-171 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:20.978546 ip-10-0-136-171 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:20.978546 ip-10-0-136-171 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:20.978546 ip-10-0-136-171 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:20.978546 ip-10-0-136-171 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:20.984323 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984222    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:20.984323 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984324    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984330    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984334    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984337    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984341    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984343    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984346    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984349    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984351    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984354    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984357    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984371    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984374    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984377    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984380    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984382    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984385    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:20.984394 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984389    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984393    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984396    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984399    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984405    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984408    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984411    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984413    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984416    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984418    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984421    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984424    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984426    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984429    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984432    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984435    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984438    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984440    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:20.984823 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984443    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984446    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984449    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984452    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984454    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984456    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984459    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984461    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984464    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984467    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984470    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984473    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984475    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984478    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984481    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984484    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984487    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984490    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984492    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984495    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:20.985289 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984497    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984500    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984503    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984505    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984508    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984510    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984513    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984515    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984518    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984522    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984526    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984529    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984535    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984538    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984540    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984543    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984545    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984548    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:20.985808 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984550    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984553    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984556    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984558    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984561    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984563    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984566    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984569    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984573    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984575    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984578    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984581    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984583    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984586    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984589    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984591    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984594    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984597    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984602    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:20.986266 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984604    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984607    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984610    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984612    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984615    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984617    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984619    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984623    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.984629    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985442    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985450    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985453    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985456    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985462    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985464    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985467    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985470    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985472    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985475    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:20.986729 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985478    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985481    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985484    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985486    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985489    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985492    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985495    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985497    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985500    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985502    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985505    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985507    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985510    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985512    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985515    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985517    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985520    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985522    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985525    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985527    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:20.987197 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985529    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985532    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985536    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985538    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985540    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985543    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985547    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985550    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985552    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985555    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985557    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985561    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985564    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985566    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985569    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985572    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985574    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985577    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985580    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985583    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:20.987675 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985586    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985590    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985595    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985598    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985601    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985603    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985606    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985608    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985611    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985614    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985617    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985619    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985622    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985624    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985627    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985630    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985632    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985635    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:20.988290 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985638    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985641    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985643    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985646    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985648    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985651    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985654    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985656    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985661    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985664    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985667    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985670    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985672    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985676    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985678    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985681    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985684    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985687    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985690    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985692    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:20.988722 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985695    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985697    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985700    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985702    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985705    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985707    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985710    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985712    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985715    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985717    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985720    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985723    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985727    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985730    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985732    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985735    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985737    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.985740    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986238    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986244    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:20.989203 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986248    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986251    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986254    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986259    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986261    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986264    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986267    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986270    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986273    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986275    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986278    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986281    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986283    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986286    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986288    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986291    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986294    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986296    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986299    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:20.989690 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986302    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986304    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986308    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986311    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986315    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986318    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986321    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986323    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986326    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986328    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986331    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986334    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986336    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986339    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986341    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986344    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986346    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986348    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986351    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:20.990200 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986354    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986356    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986359    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986361    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986364    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986367    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986369    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986372    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986374    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986377    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986380    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986382    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986385    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986388    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986391    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986393    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986396    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986401    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986403    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986406    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:20.990671 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986408    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986411    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986413    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986415    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986418    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986420    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986423    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986426    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986428    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986431    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986433    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986436    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986439    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986441    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986444    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986447    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986450    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986452    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986455    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:20.991168 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986457    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986460    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986462    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986465    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986467    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986470    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986473    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986477    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986480    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986483    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986489    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986492    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986495    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986498    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986500    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986503    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986505    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986507    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986510    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:20.991701 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986513    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986515    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986518    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986520    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986524    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986526    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.986529    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987545    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987556    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987563    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987568    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987573    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987577    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987581    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987585    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987589    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987592    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987595    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987599    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987602    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987605    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987608    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987611    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:20.992146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987615    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987619    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987623    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987626    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987629    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987632    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987636    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987640    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987643    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987646    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987649    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987652    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987655    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987658    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987662    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987665    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987669    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987672    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987675    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987678    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987681    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987684    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987689    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987692    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:20.992741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987695    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987699    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987702    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987706    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987710    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987713    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987716    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987719    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987722    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987725    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987729    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987732    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987735    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987738    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987742    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987746    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987749    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987752    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987755    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987758    2577 flags.go:64] FLAG: --help="false"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987762    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-171.ec2.internal"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987765    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987769    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:20.993349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987772    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987775    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987779    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987782    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987785    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987789    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987792    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987795    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987798    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987802    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987805    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987807    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987810    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987813    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987817    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987820    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987823    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987826    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987833    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987836    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987843    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987846    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987849    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987861    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:20.993917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987877    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987882    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987888    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987891    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987895    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987899    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987904    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987907    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987910    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987913    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987916    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987920    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987929    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987934    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987938    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987943    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987947    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987952    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987956    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987960    2577 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987964    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987968    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b98cb6316cea52fc"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987973    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987978    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987983    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987988    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:20.994505 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987994    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.987999    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988003    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988009    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988015    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988020    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988023    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988026    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988029    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988032    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988035    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988039    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988047    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988052    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988056    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988060    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988063    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988066    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988069    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988072    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988075    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988079    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988082    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988085    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988091    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:20.995115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988094    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988097    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988102    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988105    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988108    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988111    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988114    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988117    2577 flags.go:64] FLAG: --v="2"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988121    2577 flags.go:64] FLAG: --version="false"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988126    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988130    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988134    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988287    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988292    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988295    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988299    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988302    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988306    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988308    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988313    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988316    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988318    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:20.995720 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988321    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988324    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988327    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988329    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988332    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988335    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988338    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988341    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988343    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988347    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988350    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988353    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988355    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988358    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988363    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988366    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988368    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988371    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988374    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:20.996313 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988376    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988379    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988382    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988385    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988387    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988390    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988393    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988395    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988398    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988402    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988405    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988407    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988410    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988413    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988415    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988418    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988420    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988423    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988426    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988428    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:20.996777 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988431    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988435    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988438    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988440    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988443    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988446    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988449    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988452    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988454    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988456    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988459    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988462    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988464    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988467    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988469    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988472    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988475    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988478    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:20.997269 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988485    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988489    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988494    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988497    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988501    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988504    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988506    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988509    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988512    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988514    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988517    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988519    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988522    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988524    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988529    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988531    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988537    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988539    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988542    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:20.997704 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988545    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988547    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988549    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988552    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988555    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988557    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988560    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988562    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988565    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988568    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988572    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988575    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988578    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988581    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988585    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988588    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:20.998178 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988591    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.988618    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988652    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988656    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988659    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988662    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988665    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988667    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988670    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988673    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988677    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988679    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988682    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988685    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988688    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988690    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:20.998587 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988693    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988695    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988698    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988701    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988704    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988707    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988709    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988712    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988714    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988717    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988720    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988723    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988726    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988729    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988731    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988735    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988738    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988742    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988746    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:20.998975 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988748    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988751    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988754    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988757    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988759    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988763    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988765    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988769    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988772    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988774    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988777    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988780    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988782    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988785    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988788    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988791    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988793    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988797    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988800    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988802    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:20.999443 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988805    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988807    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988810    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988812    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988816    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988818    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988821    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988823    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988828    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988832    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988835    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988837    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988840    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988842    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988845    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988848    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988850    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988853    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988855    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:20.999930 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988859    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988862    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988864    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988867    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988870    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988872    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988875    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988877    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988880    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988882    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988885    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988887    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988890    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988893    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988896    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988898    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988901    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988903    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988906    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988909    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:21.000482 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988911    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988916    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988919    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988921    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988925    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988927    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988930    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988932    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988935    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988938    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988940    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988943    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988947    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.988949    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.995258    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.995274    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.995290    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.995295    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995351    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995356    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:21.000947 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995360    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995364    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995367    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995369    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995375    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995378    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995382    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995385    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995388    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995391    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995394    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995397    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995400    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995402    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995405    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995408    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995410    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995413    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:21.001488 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995416    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995419    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995421    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995424    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995427    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995429    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995432    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995435    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995439    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995442    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995445    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995447    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995450    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995453    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995455    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995458    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995460    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995463    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995465    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:21.001943 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995469    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995472    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995475    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995478    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995481    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995483    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995485    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995488    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995490    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995493    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995496    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995498    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995500    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995503    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995505    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995508    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995510    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995513    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995515    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995518    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:21.002498 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995521    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995524    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995527    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995530    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995533    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995535    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995538    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995540    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995543    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995548    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995550    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995553    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995555    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995558    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995561    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995563    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995566    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995569    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995571    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:21.002977 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995574    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995576    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995578    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995581    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995583    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995586    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995588    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995591    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995593    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995596    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995599    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995601    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995604    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995607    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995609    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995612    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995615    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995620    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995623    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:21.003707 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995625    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995628    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995630    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995632    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995635    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995637    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.995665    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995800    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995805    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995807    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995811    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995814    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995816    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995819    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995822    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995824    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:21.004181 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995827    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995830    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995832    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995835    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995837    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995840    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995842    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995845    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995849    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995851    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995854    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995856    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995859    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995862    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995865    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995867    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995870    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995872    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995875    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995877    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:21.004571 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995880    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995883    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995885    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995888    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995890    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995893    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995895    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995898    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995900    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995903    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995905    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995908    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995910    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995913    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995915    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995918    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995920    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995923    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995925    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995928    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:21.005032 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995931    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995933    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995936    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995938    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995941    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995944    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995946    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995949    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995951    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995953    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995956    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995958    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995961    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995963    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995966    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995968    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995971    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995973    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995975    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995978    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:21.005546 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995981    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995983    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995986    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995988    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995991    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995993    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995995    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.995998    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996000    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996003    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996006    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996010    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996013    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996015    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996018    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996021    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996023    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996026    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:21.006019 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996032    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996034    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996037    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996039    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996042    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996044    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996046    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996049    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996051    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996054    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996056    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996059    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996063    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996067    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996070    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996073    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996076    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:21.006521 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996079    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.996101    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996130    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996134    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996137    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996140    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996142    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996145    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996148    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996151    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996153    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996156    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996159    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996161    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996164    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996167    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:21.006949 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996169    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996172    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996174    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996177    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996179    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996181    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996184    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996186    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996189    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996191    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996194    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996196    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996199    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996202    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996204    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996207    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996209    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996212    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996215    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996217    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:21.007356 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996220    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996222    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996238    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996243    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996245    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996248    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996250    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996253    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996258    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996261    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996263    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996267    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996269    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996272    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996274    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996277    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996279    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996282    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996284    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:21.007828 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996288    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996290    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996292    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996295    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996297    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996300    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996302    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996305    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996307    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996310    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996313    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996315    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996318    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996321    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996324    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996327    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996330    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996333    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996335    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996338    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:21.008536 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996341    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996343    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996346    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996348    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996350    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996354    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996357    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996361    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996363    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996366    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996369    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996371    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996374    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996376    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996378    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996381    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996383    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996386    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996388    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:21.009036 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996390    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996393    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996396    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996398    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996401    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996403    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996405    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996408    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996412    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996416    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996420    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996422    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:20.996425    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.996474    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.998397    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.999390    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:20.999485    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:21.009502 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.000178    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:21.009939 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.001378    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:21.009939 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.005260    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:21.013208 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.013190    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:21.016312 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.016297    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:21.017115 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.017090    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:21.019778 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.019748    2577 fs.go:135] Filesystem UUIDs: map[21db0713-c6a8-4266-b88a-936b4e991d0d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cde6eb4c-f2ef-496a-8e8d-34d8610658ef:/dev/nvme0n1p3]
Oct 30 02:40:21.019862 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.019778    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:40:21.025057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.024933    2577 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:21.023534179 +0000 UTC m=+0.324789219 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098805 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f6a47c8437412f13dca2f8183bf3 SystemUUID:ec22f6a4-7c84-3741-2f13-dca2f8183bf3 BootID:9dca3f60-45e6-4953-82a3-f607a1e996b7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:1f:7b:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:1f:7b:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:92:0f:e2:76:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:21.025057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.025053    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:21.025244 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.025163    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:21.027741 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027714    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:21.027877 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027743    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:21.027924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027879    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:21.027924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027888    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:21.027924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027922    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:21.028001 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.027939    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:21.028869 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.028859    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:21.032762 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.032749    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:21.032881 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.032872    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:21.033746 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.033737    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:21.033780 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.033751    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:21.033780 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.033760    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:21.033780 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.033769    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:21.033780 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.033778    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:21.034218 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.034195    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:21.036518 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.036502    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:21.042260 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.042241    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:21.043378 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043366    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:21.043422 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043397    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:21.043561 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043551    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:21.043612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043567    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:21.043612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043573    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:21.043612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043579    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:21.043612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043585    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:21.043612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.043591    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:21.044117 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044109    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:21.044146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044118    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:21.044146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044126    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:21.044146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044131    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:21.044146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044140    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:21.044278 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044149    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:21.044795 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044776    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:21.044795 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.044787    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:21.048449 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.048435    2577 server.go:1263] "Started kubelet"
Oct 30 02:40:21.048555 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.048510    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:21.048623 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.048540    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:21.048623 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.048614    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:21.049206 ip-10-0-136-171 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:21.050280 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.049865    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:21.051893 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.051449    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:21.054878 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.054857    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:40:21.056693 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.056673    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:21.057581 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.057567    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:21.057690 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.057670    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.058124 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058109    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:21.058124 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058111    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:21.058282 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058130    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:21.058282 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058250    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:21.058282 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058259    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:21.058577 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058560    2577 factory.go:55] Registering systemd factory
Oct 30 02:40:21.058640 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058580    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:21.058783 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058767    2577 factory.go:153] Registering CRI-O factory
Oct 30 02:40:21.058783 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058783    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:21.058872 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058824    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:21.058872 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058838    2577 factory.go:103] Registering Raw factory
Oct 30 02:40:21.058872 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.058853    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:21.059798 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.059785    2577 manager.go:319] Starting recovery of all containers
Oct 30 02:40:21.071978 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.071950    2577 manager.go:324] Recovery completed
Oct 30 02:40:21.074079 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.074061    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 02:40:21.077197 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.077179    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.079300 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079275    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.079362 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079319    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.079362 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079333    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.079914 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079898    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:21.079969 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079913    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:21.079969 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079931    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:21.080054 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.079902    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.080713 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.080698    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.080771 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.080727    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.080771 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.080743    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.082140 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.082084    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-171.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:40:21.082214 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.082168    2577 policy_none.go:50] "Start"
Oct 30 02:40:21.082214 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.082184    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:21.082214 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.082197    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:21.083380 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.083357    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:21.083476 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.082078    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-171.ec2.internal.18e32e556c583a6e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-171.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-171.ec2.internal,},FirstTimestamp:2026-10-30 02:40:21.048408686 +0000 UTC m=+0.349663726,LastTimestamp:2026-10-30 02:40:21.048408686 +0000 UTC m=+0.349663726,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-171.ec2.internal,}"
Oct 30 02:40:21.083476 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.083432    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s9lkk"
Oct 30 02:40:21.084319 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.084308    2577 policy_none.go:44] "Start"
Oct 30 02:40:21.085298 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.085281    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s9lkk"
Oct 30 02:40:21.093088 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.093066    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:40:21.095386 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.095361    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:40:21.095505 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.095373    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:40:21.119844 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.119807    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:21.124323 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124302    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:21.124414 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.124336    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:21.124414 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124347    2577 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:21.124595 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124583    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:21.124663 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124598    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:21.124716 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124702    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:21.124790 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124777    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:21.124841 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.124790    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:21.127116 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.127088    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:21.127205 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.127140    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.156010 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.152949    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:21.189160 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.189138    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:21.190352 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.190337    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:21.190426 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.190363    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:21.190426 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.190390    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:21.190525 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.190429    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:40:21.204730 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.204709    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:21.224996 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.224978    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.225748 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.225731    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.225842 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.225765    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.225842 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.225780    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.225842 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.225819    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.245314 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.245271    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.245314 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.245292    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-171.ec2.internal\": node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.288482 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.288463    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.290491 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.290473    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"]
Oct 30 02:40:21.290540 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.290524    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.291412 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.291398    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.291480 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.291426    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.291480 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.291435    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.292717 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.292705    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.292870 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.292854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.292930 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.292893    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.294852 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294839    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.294924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294859    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.294924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294871    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.294924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294887    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.294924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294906    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.294924 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.294914    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.295986 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.295973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.296057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.296005    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:21.296763 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.296747    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:21.296816 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.296774    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:21.296816 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.296788    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:21.339586 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.339567    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-171.ec2.internal\" not found" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.343602 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.343586    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-171.ec2.internal\" not found" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.360006 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.359983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.360081 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.360006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.360081 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.360021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.389068 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.389048    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.460478 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.460580 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.460580 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.460580 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.460724 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460584    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.460724 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.460620    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.489332 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.489315    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.589724 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.589651    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.642879 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.642837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.646490 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.646468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:21.690597 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.690433    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.790924 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.790895    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.891526 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.891457    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.992043 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:21.992015    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:21.999176 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.999155    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:40:21.999318 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.999300    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:21.999368 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:21.999307    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:22.090277 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.090224    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:21 +0000 UTC" deadline="2028-06-13 06:48:36.10803347 +0000 UTC"
Oct 30 02:40:22.090277 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.090267    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14212h8m14.017770275s"
Oct 30 02:40:22.092562 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:22.092542    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:22.137307 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:22.137277    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b9d2d1e9934b263ff5b61e7cf24e45.slice/crio-3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a WatchSource:0}: Error finding container 3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a: Status 404 returned error can't find the container with id 3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a
Oct 30 02:40:22.144531 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.144425    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:22.150092 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:22.150065    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05848580451918f708548c7de17eb55a.slice/crio-6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2 WatchSource:0}: Error finding container 6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2: Status 404 returned error can't find the container with id 6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2
Oct 30 02:40:22.192613 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:22.192591    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:22.194452 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.194403    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal" event={"ID":"05848580451918f708548c7de17eb55a","Type":"ContainerStarted","Data":"6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2"}
Oct 30 02:40:22.195455 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.195433    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerStarted","Data":"3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a"}
Oct 30 02:40:22.207462 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.207440    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:22.208350 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.208335    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:22.208411 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.208364    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:22.208411 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.208376    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:22.284196 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.284170    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:22.292856 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:22.292833    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:40:22.328508 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.328489    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:22.359046 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.359021    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:22.385612 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.385592    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:22.385706 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.385694    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:40:22.407492 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.407441    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:22.992644 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:22.992612    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:23.035000 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.034960    2577 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:23.058349 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.058319    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:23.069258 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.069234    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:23.069725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.069705    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-8lxnt","kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal","openshift-dns/node-resolver-cqnzm","openshift-network-diagnostics/network-check-target-8xf7s","openshift-multus/multus-cjx5c","openshift-ovn-kubernetes/ovnkube-node-2cw8z","kube-system/global-pull-secret-syncer-d6zml","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572","openshift-image-registry/node-ca-xtkw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal","openshift-multus/multus-additional-cni-plugins-w97ss","kube-system/konnectivity-agent-5v56d","openshift-cluster-node-tuning-operator/tuned-bqdtf","openshift-multus/network-metrics-daemon-pkkvd"]
Oct 30 02:40:23.072209 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.072181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.072356 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.072270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:23.075026 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.075004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.077174 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.077151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.079742 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.079719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:23.079857 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.079791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:23.082162 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.082023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.085562 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.084977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.087528 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.087507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.090683 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.090195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.092713 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.092692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.092807 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.092792    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:23.093284 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.093266    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.093388 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.093270    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:40:23.093388 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.093270    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.094515 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.094492    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.094515 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.094502    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.094676 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.094583    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:40:23.095164 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.095136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.102859 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.101168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.105255 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105208    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.105404 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105385    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:23.105471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105430    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.105696 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.105775 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105737    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.105902 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105882    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:23.106021 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.105937    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:23.106088 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.106019    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.106088 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.106044    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:23.106196 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.106126    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:23.106252 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.106203    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.107810 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.107488    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:40:23.107810 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.107523    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:40:23.107810 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.107689    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:23.107810 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.107774    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.108081 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.107906    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.108139 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.108076    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:40:23.108139 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.108116    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:23.108311 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.108251    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:23.108764 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.108745    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:40:23.108871 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.108785    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.159462 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.159427    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:23.171875 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.171846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.171982 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.171888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.172057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.171991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.172057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.172147 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.172147 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172219 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172219 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.172219 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv85j\" (UniqueName: \"kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172398 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ncl\" (UniqueName: \"kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp4kp\" (UniqueName: \"kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.172721 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzfw\" (UniqueName: \"kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.172994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxcw\" (UniqueName: \"kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.173427 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cw4l\" (UniqueName: \"kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.174103 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhks\" (UniqueName: \"kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62vq9\" (UniqueName: \"kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.174749 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.175478 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.175478 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.175478 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6j5\" (UniqueName: \"kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.175478 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.173989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.175796 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.175773    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:23.175894 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.175880    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9vjm"
Oct 30 02:40:23.182702 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.182676    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9vjm"
Oct 30 02:40:23.201509 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.201478    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:23.201774 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.201755    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:23.202072 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202051    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:23.202190 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202160    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:23.202190 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:40:23.202351 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202336    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:23.202446 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202432    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:23.202446 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202441    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:23.202572 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202337    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:40:23.202572 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.202505    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:40:23.244905 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.244845    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:23.275159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv85j\" (UniqueName: \"kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.275336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ncl\" (UniqueName: \"kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4kp\" (UniqueName: \"kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.275596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzfw\" (UniqueName: \"kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxcw\" (UniqueName: \"kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.276048 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.275986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cw4l\" (UniqueName: \"kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.276609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhks\" (UniqueName: \"kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vq9\" (UniqueName: \"kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6j5\" (UniqueName: \"kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.277159 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.276941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277197    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.277912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277962    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277989    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278020    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278053    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278281    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278592    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.278682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278871    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278917    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.278976    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.279130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279147    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279228    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.277909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.279502    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.279594    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:23.779554082 +0000 UTC m=+3.080809133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.279725 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279883    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.279990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.280471 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280228    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.280235    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.280294    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:23.780277262 +0000 UTC m=+3.081532292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280385    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280453    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280658    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280730    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280928    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.280943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.281181    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.281364 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.281224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.283771 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.282761    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.283771 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.282887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.291917 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.291871    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.292167 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.292146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.300415 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.300393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv85j\" (UniqueName: \"kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.308635 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.308354    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:23.308635 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.308426    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:23.308635 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.308442    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:23.308635 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.308523    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:23.808499396 +0000 UTC m=+3.109754450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:23.311957 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.311907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzfw\" (UniqueName: \"kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.313773 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.313720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4kp\" (UniqueName: \"kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.314334 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.314309    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cw4l\" (UniqueName: \"kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.314579 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.314556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxcw\" (UniqueName: \"kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.314682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.314580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ncl\" (UniqueName: \"kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.314948 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.314866    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vq9\" (UniqueName: \"kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.315046 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.315015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6j5\" (UniqueName: \"kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.315833 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.315807    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhks\" (UniqueName: \"kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.388759 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.388729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:40:23.396770 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.396740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:40:23.401849 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.401822    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af13450_f1f2_4b72_a329_3dd739b9a10e.slice/crio-164fc277a7992a47435f5b7ecb7320794ddb10cf1bedd9a4141765b7e2e7fe20 WatchSource:0}: Error finding container 164fc277a7992a47435f5b7ecb7320794ddb10cf1bedd9a4141765b7e2e7fe20: Status 404 returned error can't find the container with id 164fc277a7992a47435f5b7ecb7320794ddb10cf1bedd9a4141765b7e2e7fe20
Oct 30 02:40:23.407910 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.407881    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8578d008_b589_4708_b51f_f94fd8176e44.slice/crio-214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51 WatchSource:0}: Error finding container 214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51: Status 404 returned error can't find the container with id 214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51
Oct 30 02:40:23.416291 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.416266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjx5c"
Oct 30 02:40:23.428139 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.428110    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f018c84_f211_4981_b33e_ad7e75ce95c9.slice/crio-f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3 WatchSource:0}: Error finding container f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3: Status 404 returned error can't find the container with id f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3
Oct 30 02:40:23.434822 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.434802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:40:23.441377 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.441355    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:23.444531 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.443108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:40:23.448348 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.448018    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a968709_cf63_4ebc_a35e_aff8293d815c.slice/crio-34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77 WatchSource:0}: Error finding container 34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77: Status 404 returned error can't find the container with id 34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77
Oct 30 02:40:23.451129 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.450948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:40:23.457948 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.457924    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc384a9_fc05_4f98_8822_4e9dbaf34282.slice/crio-a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f WatchSource:0}: Error finding container a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f: Status 404 returned error can't find the container with id a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f
Oct 30 02:40:23.463224 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.463201    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedbd56c_4ed3_4ba9_a6c6_674d0d0973f7.slice/crio-47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28 WatchSource:0}: Error finding container 47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28: Status 404 returned error can't find the container with id 47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28
Oct 30 02:40:23.467931 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.467906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:23.470049 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.470024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:40:23.484150 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:23.484129    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0ede86_75f7_4c52_ab1d_d5d6cb8747a6.slice/crio-49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa WatchSource:0}: Error finding container 49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa: Status 404 returned error can't find the container with id 49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa
Oct 30 02:40:23.780726 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.780701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:23.780821 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.780756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:23.780880 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.780867    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:23.780964 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.780863    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:23.780964 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.780919    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:24.780906948 +0000 UTC m=+4.082161977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:23.780964 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.780933    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:24.780926462 +0000 UTC m=+4.082181492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:23.881901 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:23.881862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:23.882093 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.882046    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:23.882093 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.882072    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:23.882093 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.882085    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:23.882265 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:23.882139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:24.882125654 +0000 UTC m=+4.183380683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:24.186443 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.186319    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:23 +0000 UTC" deadline="2028-03-26 15:39:25.578513956 +0000 UTC"
Oct 30 02:40:24.186443 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.186390    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12324h59m1.392128498s"
Oct 30 02:40:24.191351 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.191322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:24.191488 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.191454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:24.204249 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.204215    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal" event={"ID":"05848580451918f708548c7de17eb55a","Type":"ContainerStarted","Data":"44deb679dcbf480a009c443456dbd28add4c7b56a49334332f857e671ab37ad4"}
Oct 30 02:40:24.220182 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.220152    2577 generic.go:356] "Generic (PLEG): container finished" podID="46b9d2d1e9934b263ff5b61e7cf24e45" containerID="33a7f6d1bbbe873e802ff4e278ab413f7899099cc1d655d8721f12f639d773a1" exitCode=0
Oct 30 02:40:24.220470 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.220444    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerDied","Data":"33a7f6d1bbbe873e802ff4e278ab413f7899099cc1d655d8721f12f639d773a1"}
Oct 30 02:40:24.226937 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.226911    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerStarted","Data":"49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa"}
Oct 30 02:40:24.241950 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.241921    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572" event={"ID":"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7","Type":"ContainerStarted","Data":"47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28"}
Oct 30 02:40:24.259991 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.259932    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal" podStartSLOduration=1.259913633 podStartE2EDuration="1.259913633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:24.259743456 +0000 UTC m=+3.560998511" watchObservedRunningTime="2026-10-30 02:40:24.259913633 +0000 UTC m=+3.561168688"
Oct 30 02:40:24.261596 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.261566    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"0e7a69e7612e387834aac81ee8d523e8aaabef16d664d3a20c7384686774a522"}
Oct 30 02:40:24.271553 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.271520    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtkw9" event={"ID":"0a968709-cf63-4ebc-a35e-aff8293d815c","Type":"ContainerStarted","Data":"34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77"}
Oct 30 02:40:24.273555 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.273507    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqnzm" event={"ID":"8578d008-b589-4708-b51f-f94fd8176e44","Type":"ContainerStarted","Data":"214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51"}
Oct 30 02:40:24.275088 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.275002    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bqdtf" event={"ID":"5af13450-f1f2-4b72-a329-3dd739b9a10e","Type":"ContainerStarted","Data":"164fc277a7992a47435f5b7ecb7320794ddb10cf1bedd9a4141765b7e2e7fe20"}
Oct 30 02:40:24.287155 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.287124    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v56d" event={"ID":"f669d42f-1e2c-43c2-8172-202d70c39f0d","Type":"ContainerStarted","Data":"cf28a82a3f43d61879875907b55b442deb51a7f61a4a3e7f4af323cb3eccc6f9"}
Oct 30 02:40:24.297365 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.297334    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8lxnt" event={"ID":"7dc384a9-fc05-4f98-8822-4e9dbaf34282","Type":"ContainerStarted","Data":"a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f"}
Oct 30 02:40:24.312093 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.312062    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjx5c" event={"ID":"1f018c84-f211-4981-b33e-ad7e75ce95c9","Type":"ContainerStarted","Data":"f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3"}
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.795580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.795929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.796077    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.796141    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.796122078 +0000 UTC m=+6.097377125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.796209    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:24.796709 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.796243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.796230721 +0000 UTC m=+6.097485766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:24.896694 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:24.896657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:24.896877 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.896860    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:24.896949 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.896887    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:24.896949 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.896901    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:24.897055 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:24.896975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.896953026 +0000 UTC m=+6.198208082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:25.187273 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:25.186734    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:23 +0000 UTC" deadline="2028-06-17 02:34:44.103747587 +0000 UTC"
Oct 30 02:40:25.187273 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:25.186792    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14303h54m18.916959942s"
Oct 30 02:40:25.192907 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:25.192387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:25.192907 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:25.192527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:25.193397 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:25.193216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:25.193397 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:25.193346    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:25.336745 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:25.336712    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerStarted","Data":"14ac0da8c88a3340bbe907f2979d5a5bb6cedf0faa1ad9dac4cfe07aae5b253c"}
Oct 30 02:40:26.190604 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.190568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:26.190992 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.190742    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:26.355411 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.355375    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerStarted","Data":"00c488e7607aee04e96b5c5332487261c8523f76003a2e3154f424a2073ce01a"}
Oct 30 02:40:26.436555 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.435905    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" podStartSLOduration=3.435885332 podStartE2EDuration="3.435885332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:25.381339825 +0000 UTC m=+4.682594879" watchObservedRunningTime="2026-10-30 02:40:26.435885332 +0000 UTC m=+5.737140384"
Oct 30 02:40:26.810374 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.810272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:26.810374 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.810361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:26.810620 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.810515    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:26.810620 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.810579    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.810561471 +0000 UTC m=+10.111816522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:26.811076 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.810991    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:26.811076 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.811044    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.811028807 +0000 UTC m=+10.112283851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:26.911145 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:26.911108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:26.911371 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.911353    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:26.911449 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.911378    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:26.911449 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.911392    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:26.911573 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:26.911452    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.911432139 +0000 UTC m=+10.212687193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:27.193304 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:27.191379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:27.193304 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:27.191525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:27.193304 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:27.191913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:27.193304 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:27.192011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:27.364110 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:27.364077    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="00c488e7607aee04e96b5c5332487261c8523f76003a2e3154f424a2073ce01a" exitCode=0
Oct 30 02:40:27.364232 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:27.364131    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"00c488e7607aee04e96b5c5332487261c8523f76003a2e3154f424a2073ce01a"}
Oct 30 02:40:28.191013 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:28.190972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:28.191194 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:28.191124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:28.372732 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:28.372062    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v56d" event={"ID":"f669d42f-1e2c-43c2-8172-202d70c39f0d","Type":"ContainerStarted","Data":"c1311c0055b482644936b97c324df405af8c850781091b4bac7b0e50b50ce132"}
Oct 30 02:40:28.417562 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:28.417502    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5v56d" podStartSLOduration=2.535814578 podStartE2EDuration="6.417482786s" totalImagesPullingTime="3.881668208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:28.416810039 +0000 UTC m=+7.718065094" watchObservedRunningTime="2026-10-30 02:40:28.417482786 +0000 UTC m=+7.718737877"
Oct 30 02:40:29.192386 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:29.191609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:29.192386 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:29.191752    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:29.192386 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:29.192203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:29.192386 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:29.192337    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:29.377716 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:29.377675    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572" event={"ID":"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7","Type":"ContainerStarted","Data":"8a03c7552da50c5b02f858a9fba9e74e94c2cc866134ac4ea61d20d48fdd389a"}
Oct 30 02:40:29.381092 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:29.381058    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtkw9" event={"ID":"0a968709-cf63-4ebc-a35e-aff8293d815c","Type":"ContainerStarted","Data":"adb0db413ef772db7feeb1a1b94ae2ca29047d223e231cd97679233a11bdd6d4"}
Oct 30 02:40:29.424685 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:29.424631    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xtkw9" podStartSLOduration=2.244619308 podStartE2EDuration="7.424610381s" totalImagesPullingTime="5.179991073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.424315337 +0000 UTC m=+8.725570391" watchObservedRunningTime="2026-10-30 02:40:29.424610381 +0000 UTC m=+8.725865434"
Oct 30 02:40:30.191094 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:30.190990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:30.191265 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.191127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:30.798918 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:30.798885    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:30.844012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:30.844055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.844165    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.844227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:38.84420841 +0000 UTC m=+18.145463459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.844301    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:30.844410 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.844327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:38.844318953 +0000 UTC m=+18.145573996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:30.955295 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:30.952466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:30.955295 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.952688    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:30.955295 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.952708    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:30.955295 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.952719    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:30.955295 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:30.952800    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:38.952778512 +0000 UTC m=+18.254033545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:31.129892 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.128731    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:30.798910443Z","UUID":"98056559-e884-4cf7-b821-968abb8ccc3b","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:31.136126 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.131246    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:31.136126 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.131280    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:31.193977 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.193934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:31.194163 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:31.194062    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:31.194447 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.194429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:31.194545 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:31.194520    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:31.389428 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.389347    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:31.389719 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.389696    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:31.392610 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:31.392575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572" event={"ID":"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7","Type":"ContainerStarted","Data":"279dca18c24d4aa7267dad74ae880d267e5656249a00eefff50d3e7256e16b89"}
Oct 30 02:40:32.191423 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.191125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:32.191891 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:32.191488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:32.398274 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.398138    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="59e9c9d444761cd997fd20e36b02e0876bb9f3aa1b4bef9b9c7a0b15ef3d4710" exitCode=0
Oct 30 02:40:32.398510 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.398443    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"59e9c9d444761cd997fd20e36b02e0876bb9f3aa1b4bef9b9c7a0b15ef3d4710"}
Oct 30 02:40:32.401499 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.401466    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572" event={"ID":"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7","Type":"ContainerStarted","Data":"a2b0e34e1e5b3403c2c1f4aaa083f908f38bac1e91c1a80e4fe4a9b4d3774b80"}
Oct 30 02:40:32.403594 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.403563    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cqnzm" event={"ID":"8578d008-b589-4708-b51f-f94fd8176e44","Type":"ContainerStarted","Data":"a147f953042274a70d76cd3cdc2e6a0e710f75e5c65e0e6277dd978ebd215331"}
Oct 30 02:40:32.406222 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.406196    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjx5c" event={"ID":"1f018c84-f211-4981-b33e-ad7e75ce95c9","Type":"ContainerStarted","Data":"8de2837f258017652d58331c9866f060dd23fd78cebe633cf3214dedfe8cde10"}
Oct 30 02:40:32.492565 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.492458    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572" podStartSLOduration=2.472992103 podStartE2EDuration="10.492441869s" totalImagesPullingTime="8.019449766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.49212531 +0000 UTC m=+11.793380374" watchObservedRunningTime="2026-10-30 02:40:32.492441869 +0000 UTC m=+11.793696918"
Oct 30 02:40:32.538192 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.538138    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cjx5c" podStartSLOduration=2.565103058 podStartE2EDuration="10.538119462s" totalImagesPullingTime="7.973016404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.537856188 +0000 UTC m=+11.839111241" watchObservedRunningTime="2026-10-30 02:40:32.538119462 +0000 UTC m=+11.839374520"
Oct 30 02:40:32.580982 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:32.580916    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cqnzm" podStartSLOduration=2.368157397 podStartE2EDuration="10.580893907s" totalImagesPullingTime="8.21273651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.580605859 +0000 UTC m=+11.881860913" watchObservedRunningTime="2026-10-30 02:40:32.580893907 +0000 UTC m=+11.882148961"
Oct 30 02:40:33.191464 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.191433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:33.191941 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:33.191589    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:33.191941 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.191609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:33.191941 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:33.191719    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:33.410458 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.410423    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="fb4bbbdb05aa5c1fcefc2b0bbda7173ccb72cbf7e08413b6016bc149870a0193" exitCode=0
Oct 30 02:40:33.410610 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.410476    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"fb4bbbdb05aa5c1fcefc2b0bbda7173ccb72cbf7e08413b6016bc149870a0193"}
Oct 30 02:40:33.412526 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.412369    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8lxnt" event={"ID":"7dc384a9-fc05-4f98-8822-4e9dbaf34282","Type":"ContainerStarted","Data":"3379fa6a34a4b6fdc11d1605ac9d049cb46c4b72010d717246de9587e2534cba"}
Oct 30 02:40:33.504196 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:33.504147    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8lxnt" podStartSLOduration=3.186935305 podStartE2EDuration="11.504133912s" totalImagesPullingTime="8.317198607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:33.50387061 +0000 UTC m=+12.805125660" watchObservedRunningTime="2026-10-30 02:40:33.504133912 +0000 UTC m=+12.805388962"
Oct 30 02:40:34.190759 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:34.190732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:34.190883 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:34.190845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:34.417067 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:34.416986    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="d59710682bbc9f357987584a8c7eb0d936d1fa5b89d95e97e158c61539e8dbb4" exitCode=0
Oct 30 02:40:34.417699 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:34.417059    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"d59710682bbc9f357987584a8c7eb0d936d1fa5b89d95e97e158c61539e8dbb4"}
Oct 30 02:40:34.419197 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:34.419123    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bqdtf" event={"ID":"5af13450-f1f2-4b72-a329-3dd739b9a10e","Type":"ContainerStarted","Data":"7e8456be0a78a428c7ee6ad61eae2bab06710179f63bb06eba78920ae7e3df9f"}
Oct 30 02:40:34.506754 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:34.506703    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bqdtf" podStartSLOduration=1.797809258 podStartE2EDuration="12.506688906s" totalImagesPullingTime="10.708879648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:34.506558284 +0000 UTC m=+13.807813336" watchObservedRunningTime="2026-10-30 02:40:34.506688906 +0000 UTC m=+13.807943955"
Oct 30 02:40:35.191276 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:35.191247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:35.191449 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:35.191379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:35.191449 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:35.191429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:35.191602 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:35.191573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:36.190786 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:36.190758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:36.191509 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:36.190874    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:37.191605 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:37.191381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:37.192099 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:37.191429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:37.192099 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:37.191720    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:37.192099 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:37.191873    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:37.429405 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:37.429367    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="a6c813a7091bb59bfe1677c132115143356f50a50b64a7dcf7249981f8a81c0f" exitCode=0
Oct 30 02:40:37.429572 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:37.429412    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"a6c813a7091bb59bfe1677c132115143356f50a50b64a7dcf7249981f8a81c0f"}
Oct 30 02:40:38.191511 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:38.191474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:38.191646 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:38.191574    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:38.434359 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:38.434324    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="183fa1c0a9046968981c0ec85a74ae1aa0b4fc90d5747edb5f3a231ada06e9e1" exitCode=0
Oct 30 02:40:38.434539 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:38.434372    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerDied","Data":"183fa1c0a9046968981c0ec85a74ae1aa0b4fc90d5747edb5f3a231ada06e9e1"}
Oct 30 02:40:38.910158 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:38.909990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:38.910336 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:38.910179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:38.910336 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:38.910135    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:38.910336 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:38.910274    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:38.910336 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:38.910311    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret podName:81e5958e-50f9-4bde-948e-841d12e07b7d nodeName:}" failed. No retries permitted until 2026-10-30 02:40:54.910293923 +0000 UTC m=+34.211548982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret") pod "global-pull-secret-syncer-d6zml" (UID: "81e5958e-50f9-4bde-948e-841d12e07b7d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:38.910336 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:38.910331    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:54.910322605 +0000 UTC m=+34.211577632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:39.011146 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.011047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:39.011309 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.011213    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:39.011309 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.011248    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:39.011309 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.011258    2577 projected.go:196] Error preparing data for projected volume kube-api-access-bb7cw for pod openshift-network-diagnostics/network-check-target-8xf7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:39.011395 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.011313    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw podName:c811b314-2fff-49fd-abc8-0a8304fb94e6 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:55.011296598 +0000 UTC m=+34.312551626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb7cw" (UniqueName: "kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw") pod "network-check-target-8xf7s" (UID: "c811b314-2fff-49fd-abc8-0a8304fb94e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:39.191223 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.191180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:39.191399 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.191180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:39.191399 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.191342    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:39.191530 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:39.191445    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:39.439208 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.439170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w97ss" event={"ID":"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6","Type":"ContainerStarted","Data":"6f74c4841904887f823424da656aeb2a5755ef643425154018bd0afeabaca71e"}
Oct 30 02:40:39.441943 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.441922    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"290b178058e22fff6aae5282d01e29e7c3d426f1c8f7d60c9300a3355ddfb882"}
Oct 30 02:40:39.442034 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.441975    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"7b2acc5e745880218d0988cb4435435632a1e91c0038e3576cc60337a5f10816"}
Oct 30 02:40:39.442034 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.442004    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"d108a6d3b0427c91a22be4655c28d5938ce78b91ce1b43f3040396af0d5ba3d8"}
Oct 30 02:40:39.442034 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.442030    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"ebf4f96d346f879be13f230f615dd5c973eb7e2e104c7715eceebc6e64901406"}
Oct 30 02:40:39.442126 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.442055    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"c62333db26319040b49536eee6ea2f98156f5081cf53a77814623cf1cabaaacf"}
Oct 30 02:40:39.442126 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:39.442082    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"d83db4e9708284874ce0500fc02421471ef2466d322375363e37965556a21afb"}
Oct 30 02:40:40.191300 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:40.191265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:40.191466 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:40.191363    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:41.192663 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:41.192578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:41.192663 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:41.192628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:41.193166 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:41.192701    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:41.193166 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:41.192805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:41.448997 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:41.448911    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"2f0eb1dc838c222cb991596fe2a0a32d0b6979bf1ca017503cbbebb40f30b1a5"}
Oct 30 02:40:42.190997 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:42.190957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:42.191189 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:42.191077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:43.191631 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.191563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:43.192264 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:43.191662    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:43.192264 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.191701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:43.192264 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:43.191756    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:43.456567 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.456493    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" event={"ID":"f1de0f3c-4516-4210-afaa-3657c54257af","Type":"ContainerStarted","Data":"7a312f2711808ab0337b176dbe91e41c0a2d61ec7a109051ba317835b7fa8687"}
Oct 30 02:40:43.456781 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.456763    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:43.456852 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.456835    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:43.471682 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.471655    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:43.550390 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.550340    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z" podStartSLOduration=6.554672937 podStartE2EDuration="21.550327044s" totalImagesPullingTime="14.995654107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:43.536083396 +0000 UTC m=+22.837338444" watchObservedRunningTime="2026-10-30 02:40:43.550327044 +0000 UTC m=+22.851582092"
Oct 30 02:40:43.550632 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:43.550612    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w97ss" podStartSLOduration=11.146237454 podStartE2EDuration="21.550607064s" totalImagesPullingTime="10.40436961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:39.484647123 +0000 UTC m=+18.785902173" watchObservedRunningTime="2026-10-30 02:40:43.550607064 +0000 UTC m=+22.851862116"
Oct 30 02:40:44.191409 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.191379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:44.191589 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:44.191503    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:44.458489 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.458414    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:44.472896 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.472866    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:40:44.988649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.988403    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6zml"]
Oct 30 02:40:44.988906 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.988724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:44.988906 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:44.988832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:44.988906 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.988871    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8xf7s"]
Oct 30 02:40:44.989057 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.988954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:44.989057 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:44.989017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:44.991877 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.991852    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pkkvd"]
Oct 30 02:40:44.991983 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:44.991960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:44.992094 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:44.992076    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:46.190781 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:46.190748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:46.191264 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:46.190787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:46.191264 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:46.190789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:46.191264 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:46.190860    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:46.191264 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:46.190977    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:46.191264 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:46.191067    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:47.463213 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:47.463181    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:47.463696 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:47.463346    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:47.463907 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:47.463888    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:40:48.191408 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:48.191367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:48.191408 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:48.191389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:48.191663 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:48.191476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:48.191663 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:48.191487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:48.191663 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:48.191563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:48.191663 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:48.191648    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:50.191648 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:50.191615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:50.192075 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:50.191676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:50.192075 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:50.191700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:50.192075 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:50.191786    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6zml" podUID="81e5958e-50f9-4bde-948e-841d12e07b7d"
Oct 30 02:40:50.192075 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:50.191812    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xf7s" podUID="c811b314-2fff-49fd-abc8-0a8304fb94e6"
Oct 30 02:40:50.192075 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:50.191891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pkkvd" podUID="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6"
Oct 30 02:40:50.991305 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:50.991282    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeReady"
Oct 30 02:40:50.991454 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:50.991406    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:40:51.059457 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.059431    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7k44l"]
Oct 30 02:40:51.061847 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.061832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.070274 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.070001    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khc7g"]
Oct 30 02:40:51.073023 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.073004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.076795 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.076774    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c4grk"]
Oct 30 02:40:51.079779 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.079763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.086571 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.086526    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:40:51.086678 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.086589    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xn2tj\""
Oct 30 02:40:51.087884 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.087864    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7k44l"]
Oct 30 02:40:51.091136 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.091115    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:40:51.091136 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.091118    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:40:51.099617 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.099595    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khc7g"]
Oct 30 02:40:51.105240 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.105206    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c4grk"]
Oct 30 02:40:51.185220 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185198    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:40:51.185387 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185352    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:40:51.185387 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185359    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:40:51.185387 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185380    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:40:51.185585 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185570    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:40:51.185585 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185570    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:40:51.185658 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185577    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdxrj\""
Oct 30 02:40:51.185658 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.185620    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd75s\""
Oct 30 02:40:51.194803 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz9l\" (UniqueName: \"kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.194803 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.194981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.195004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc6l\" (UniqueName: \"kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.195055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.195105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.195210 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.195121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4ql\" (UniqueName: \"kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.296415 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.296415 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.296523    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc6l\" (UniqueName: \"kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.296600    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:51.79658079 +0000 UTC m=+31.097835834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.296649 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4ql\" (UniqueName: \"kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz9l\" (UniqueName: \"kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.296620    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.296830    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls podName:701387a9-97f7-42f7-b9f8-11a044f5a3c7 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:51.796815805 +0000 UTC m=+31.098070846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls") pod "dns-default-khc7g" (UID: "701387a9-97f7-42f7-b9f8-11a044f5a3c7") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.296897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.296651    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.297026    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:51.797014227 +0000 UTC m=+31.098269254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:51.297073 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.297061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.297486 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.297351    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.300568 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.300541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.327436 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.327371    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4ql\" (UniqueName: \"kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.327668 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.327648    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc6l\" (UniqueName: \"kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.327761 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.327739    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz9l\" (UniqueName: \"kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.375673 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.375594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:40:51.688083 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.687931    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7k44l"]
Oct 30 02:40:51.690968 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:51.690938    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1acad5_796c_4ea8_b791_ce810c9438dd.slice/crio-b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c WatchSource:0}: Error finding container b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c: Status 404 returned error can't find the container with id b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c
Oct 30 02:40:51.800871 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.800835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.801029 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.800898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:51.801029 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:51.800915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:51.801029 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.800991    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:51.801029 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.801010    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.801029 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.801015    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.801177 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.801066    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls podName:701387a9-97f7-42f7-b9f8-11a044f5a3c7 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:52.801050265 +0000 UTC m=+32.102305294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls") pod "dns-default-khc7g" (UID: "701387a9-97f7-42f7-b9f8-11a044f5a3c7") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.801177 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.801084    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:52.801075644 +0000 UTC m=+32.102330671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.801177 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:51.801099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:52.801091091 +0000 UTC m=+32.102346118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:52.191351 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.191311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:52.191351 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.191339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:52.191609 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.191327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:52.208623 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208597    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208597    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208598    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208674    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208706    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208730    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:40:52.209272 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.208787    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:40:52.476484 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.476407    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7k44l" event={"ID":"3e1acad5-796c-4ea8-b791-ce810c9438dd","Type":"ContainerStarted","Data":"b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c"}
Oct 30 02:40:52.808623 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.808581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.808633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:52.808681    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.808695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:52.808754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:54.808734897 +0000 UTC m=+34.109989937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:52.808800    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:52.808851 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:52.808851    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:54.80883665 +0000 UTC m=+34.110091694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:52.812451 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.812424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:52.885821 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.885791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:52.905849 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:52.905822    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqnzm_8578d008-b589-4708-b51f-f94fd8176e44/dns-node-resolver/0.log"
Oct 30 02:40:53.200619 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:53.200544    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khc7g"]
Oct 30 02:40:53.203159 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:53.203130    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701387a9_97f7_42f7_b9f8_11a044f5a3c7.slice/crio-e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350 WatchSource:0}: Error finding container e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350: Status 404 returned error can't find the container with id e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350
Oct 30 02:40:53.480130 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:53.480049    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc7g" event={"ID":"701387a9-97f7-42f7-b9f8-11a044f5a3c7","Type":"ContainerStarted","Data":"e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350"}
Oct 30 02:40:54.048465 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.048436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtkw9_0a968709-cf63-4ebc-a35e-aff8293d815c/node-ca/0.log"
Oct 30 02:40:54.824037 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.823990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:54.824586 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.824082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:54.824586 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:54.824274    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:54.824586 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:54.824381    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:58.824357509 +0000 UTC m=+38.125612542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:54.824727 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.824660    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:54.925608 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.925366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:54.925780 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.925698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:40:54.925874 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:54.925852    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:54.925949 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:54.925938    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs podName:0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:26.925917899 +0000 UTC m=+66.227172942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs") pod "network-metrics-daemon-pkkvd" (UID: "0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6") : secret "metrics-daemon-secret" not found
Oct 30 02:40:54.928345 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:54.928319    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:55.026145 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.026105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:55.028510 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.028485    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:55.157310 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.157204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v56d_f669d42f-1e2c-43c2-8172-202d70c39f0d/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:55.222185 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.222148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:40:55.242087 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.242060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:55.488188 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.488155    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7k44l" event={"ID":"3e1acad5-796c-4ea8-b791-ce810c9438dd","Type":"ContainerStarted","Data":"7289f714e31decbf27bfcfeefb9f9870264779428969b76c95b76adab3ab4abf"}
Oct 30 02:40:55.489814 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.489779    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc7g" event={"ID":"701387a9-97f7-42f7-b9f8-11a044f5a3c7","Type":"ContainerStarted","Data":"381f58b5516caffd78ad896041a29b60d95c97d710405f477965ba4de221f540"}
Oct 30 02:40:55.530735 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.530686    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7k44l" podStartSLOduration=1.709412503 podStartE2EDuration="4.530669833s" totalImagesPullingTime="2.82125733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:55.529863329 +0000 UTC m=+34.831118377" watchObservedRunningTime="2026-10-30 02:40:55.530669833 +0000 UTC m=+34.831924885"
Oct 30 02:40:55.554141 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.554120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v56d_f669d42f-1e2c-43c2-8172-202d70c39f0d/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:55.619593 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.619568    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8xf7s"]
Oct 30 02:40:55.622147 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:55.622108    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc811b314_2fff_49fd_abc8_0a8304fb94e6.slice/crio-f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5 WatchSource:0}: Error finding container f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5: Status 404 returned error can't find the container with id f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5
Oct 30 02:40:55.632382 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.632302    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6zml"]
Oct 30 02:40:55.634679 ip-10-0-136-171 kubenswrapper[2577]: W1030 02:40:55.634651    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e5958e_50f9_4bde_948e_841d12e07b7d.slice/crio-e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586 WatchSource:0}: Error finding container e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586: Status 404 returned error can't find the container with id e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586
Oct 30 02:40:55.698866 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:55.698792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v56d_f669d42f-1e2c-43c2-8172-202d70c39f0d/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:56.495158 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:56.495124    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc7g" event={"ID":"701387a9-97f7-42f7-b9f8-11a044f5a3c7","Type":"ContainerStarted","Data":"62a6d8b233f319ba41b0be0f5d4fc979536315d313e9d480b7de861a3514ab3b"}
Oct 30 02:40:56.495636 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:56.495271    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khc7g"
Oct 30 02:40:56.496836 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:56.496808    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6zml" event={"ID":"81e5958e-50f9-4bde-948e-841d12e07b7d","Type":"ContainerStarted","Data":"e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586"}
Oct 30 02:40:56.498240 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:56.498202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8xf7s" event={"ID":"c811b314-2fff-49fd-abc8-0a8304fb94e6","Type":"ContainerStarted","Data":"f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5"}
Oct 30 02:40:58.857086 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:58.857052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:40:58.857599 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:58.857188    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:58.857599 ip-10-0-136-171 kubenswrapper[2577]: E1030 02:40:58.857273    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls podName:3972a4c2-785f-4c36-967d-c3eb4bdd548e nodeName:}" failed. No retries permitted until 2026-10-30 02:41:06.857251369 +0000 UTC m=+46.158506414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c4grk" (UID: "3972a4c2-785f-4c36-967d-c3eb4bdd548e") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:59.508633 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:59.508414    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8xf7s" event={"ID":"c811b314-2fff-49fd-abc8-0a8304fb94e6","Type":"ContainerStarted","Data":"26bba50b335daec21b308ab5234ef58af6f2c22c5dfdfd902f351c94f1075e86"}
Oct 30 02:40:59.508850 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:59.508664    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:40:59.554461 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:59.554399    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-khc7g" podStartSLOduration=6.441634733 podStartE2EDuration="8.5543814s" totalImagesPullingTime="2.112746667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:56.539201666 +0000 UTC m=+35.840456717" watchObservedRunningTime="2026-10-30 02:40:59.5543814 +0000 UTC m=+38.855636449"
Oct 30 02:40:59.554912 ip-10-0-136-171 kubenswrapper[2577]: I1030 02:40:59.554877    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8xf7s" podStartSLOduration=34.219440001 podStartE2EDuration="37.554866911s" totalImagesPullingTime="3.33542691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:59.553928413 +0000 UTC m=+38.855183463" watchObservedRunningTime="2026-10-30 02:40:59.554866911 +0000 UTC m=+38.856121962"
Oct 30 02:41:00.862002 ip-10-0-136-171 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:41:00.871833 ip-10-0-136-171 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:41:00.872043 ip-10-0-136-171 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:41:00.872247 ip-10-0-136-171 systemd[1]: kubelet.service: Consumed 2.304s CPU time.
Oct 30 02:41:00.882881 ip-10-0-136-171 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:41:00.964015 ip-10-0-136-171 kubenswrapper[4214]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:00.964015 ip-10-0-136-171 kubenswrapper[4214]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:41:00.964015 ip-10-0-136-171 kubenswrapper[4214]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:00.964728 ip-10-0-136-171 kubenswrapper[4214]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:00.966041 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.965974    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.966041 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966041    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966044    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966047    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966050    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966054    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966057    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966060    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966063    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966066    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966069    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966072    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966075    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966077    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966080    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966082    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966085    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966087    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966090    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966093    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.966120 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966095    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966098    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966102    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966107    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966110    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966113    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966116    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966124    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966127    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966130    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966133    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966136    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966139    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966141    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966144    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966146    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966149    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966152    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966154    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966157    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.966610 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966159    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966162    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966164    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966167    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966169    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966172    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966175    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966177    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966180    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966182    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966185    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966187    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966190    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966193    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966197    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966200    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966202    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966205    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966208    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966210    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.967114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966213    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966215    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966218    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966221    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966223    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966239    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966242    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966244    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966247    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966250    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966252    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966255    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966258    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966262    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966266    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966270    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966273    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966276    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966279    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.967620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966281    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966286    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966288    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966291    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966294    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966296    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966299    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966301    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966304    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966307    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966309    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966312    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966314    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966317    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966319    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966322    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966324    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966327    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966329    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966332    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.968103 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966334    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966337    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966339    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966343    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966346    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966348    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.966351    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967170    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967177    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967181    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967184    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967187    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967191    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967197    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967200    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967202    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967205    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967208    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967211    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967213    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.968602 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967216    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967218    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967221    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967224    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967240    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967243    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967246    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967249    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967252    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967254    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967257    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967260    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967262    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967265    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967267    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967270    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967273    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967275    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967278    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967280    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.969108 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967283    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967285    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967288    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967291    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967293    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967303    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967306    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967309    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967311    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967314    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967317    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967319    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967322    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967324    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967327    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967330    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967332    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967335    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967339    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967342    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.969620 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967345    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967348    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967351    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967353    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967356    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967358    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967361    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967363    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967366    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967368    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967371    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967374    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967376    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967378    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967381    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967384    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967387    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967392    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967395    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.970145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967397    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967400    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967403    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967405    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967408    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967411    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967414    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967416    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967419    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967422    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967425    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967428    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967430    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967433    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967435    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967438    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967440    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967443    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967446    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967448    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.970637 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967451    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967455    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967459    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967461    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967464    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967466    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967469    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967472    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967475    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967477    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967483    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967485    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967489    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967491    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967951    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967956    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967959    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967962    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967965    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.971176 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967968    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967970    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967973    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967975    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967978    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967981    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967983    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967986    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967989    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967991    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967994    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967996    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.967999    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968002    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968006    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968010    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968013    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968016    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968018    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968021    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.971645 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968024    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968026    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968029    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968035    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968037    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968040    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968043    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968045    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968048    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968051    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968053    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968056    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968058    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968061    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968063    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968066    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968068    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968071    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968074    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.972315 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968076    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968079    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968081    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968084    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968086    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968089    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968091    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968094    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968097    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968099    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968102    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968106    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968109    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968112    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968115    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968117    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968121    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968124    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968127    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.972822 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968130    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968133    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968136    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968139    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968141    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968144    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968146    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968149    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968151    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968154    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968156    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968159    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968162    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968164    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968167    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968169    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968172    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968174    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968177    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968179    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.973312 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968182    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968184    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968187    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968190    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968193    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968195    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968198    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968200    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968203    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968207    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968210    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968212    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968215    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968218    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968221    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968223    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968239    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968242    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968244    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.974013 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968247    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968250    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968252    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.968255    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968347    4214 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968358    4214 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968377    4214 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968384    4214 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968388    4214 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968392    4214 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968396    4214 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968400    4214 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968403    4214 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968406    4214 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968410    4214 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968413    4214 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968416    4214 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968420    4214 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968423    4214 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968426    4214 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968429    4214 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968432    4214 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968437    4214 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968440    4214 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:41:00.974703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968443    4214 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968446    4214 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968450    4214 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968454    4214 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968457    4214 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968460    4214 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968463    4214 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968466    4214 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968469    4214 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968472    4214 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968475    4214 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968478    4214 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968482    4214 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968486    4214 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968488    4214 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968491    4214 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968494    4214 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968497    4214 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968502    4214 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968505    4214 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968508    4214 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968511    4214 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968514    4214 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968518    4214 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:41:00.975315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968523    4214 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968526    4214 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968529    4214 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968532    4214 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968535    4214 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968538    4214 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968542    4214 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968545    4214 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968548    4214 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968551    4214 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968555    4214 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968558    4214 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968562    4214 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968565    4214 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968568    4214 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968571    4214 flags.go:64] FLAG: --help="false"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968574    4214 flags.go:64] FLAG: --hostname-override="ip-10-0-136-171.ec2.internal"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968577    4214 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968580    4214 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968583    4214 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968586    4214 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968590    4214 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968592    4214 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:41:00.975905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968595    4214 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968598    4214 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968601    4214 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968604    4214 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968607    4214 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968610    4214 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968613    4214 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968616    4214 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968619    4214 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968623    4214 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968626    4214 flags.go:64] FLAG: --lock-file=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968629    4214 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968632    4214 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968635    4214 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968640    4214 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968644    4214 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968647    4214 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968650    4214 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968653    4214 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968656    4214 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968659    4214 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968662    4214 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968667    4214 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968670    4214 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968674    4214 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:41:00.976518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968677    4214 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968680    4214 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968683    4214 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968686    4214 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968688    4214 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968691    4214 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968694    4214 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968701    4214 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968704    4214 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968707    4214 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968710    4214 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968712    4214 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968715    4214 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968718    4214 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968721    4214 flags.go:64] FLAG: --port="10250"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968724    4214 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968729    4214 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b98cb6316cea52fc"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968732    4214 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968735    4214 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968738    4214 flags.go:64] FLAG: --register-node="true"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968741    4214 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968745    4214 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968748    4214 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968751    4214 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968754    4214 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968758    4214 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:41:00.977115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968761    4214 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968764    4214 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968767    4214 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968770    4214 flags.go:64] FLAG: --runonce="false"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968774    4214 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968777    4214 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968780    4214 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968782    4214 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968785    4214 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968788    4214 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968791    4214 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968794    4214 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968797    4214 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968800    4214 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968803    4214 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968806    4214 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968809    4214 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968812    4214 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968814    4214 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968819    4214 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968822    4214 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968825    4214 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968830    4214 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968833    4214 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968836    4214 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:41:00.977888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968839    4214 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968842    4214 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968845    4214 flags.go:64] FLAG: --v="2"
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968849    4214 flags.go:64] FLAG: --version="false"
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968853    4214 flags.go:64] FLAG: --vmodule=""
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968857    4214 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.968860    4214 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969019    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969024    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969027    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969030    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969032    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969042    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969045    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969048    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969050    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969053    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969057    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969061    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969065    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969068    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.978518 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969070    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969073    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969076    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969078    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969081    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969083    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969086    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969089    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969093    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969097    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969101    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969104    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969107    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969109    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969112    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969115    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969118    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969120    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969123    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.979039 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969126    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969128    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969131    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969134    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969136    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969145    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969148    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969150    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969153    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969155    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969158    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969161    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969163    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969166    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969168    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969171    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969174    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969176    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969179    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.979573 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969183    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969185    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969189    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969192    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969195    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969197    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969200    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969202    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969205    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969207    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969210    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969212    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969215    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969217    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969220    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969223    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969238    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969241    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969244    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.980040 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969252    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969255    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969257    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969260    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969263    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969266    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969268    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969271    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969274    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969276    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969279    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969281    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969284    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969287    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969290    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969293    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969296    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969298    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969301    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969303    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.980521 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969306    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969309    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969311    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969314    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969316    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969319    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969322    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969324    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969327    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969329    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969332    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969334    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969337    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969345    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969348    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.969375    4214 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:00.981046 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969415    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969420    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969423    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969426    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969428    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969431    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969433    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969436    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969438    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969441    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969446    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969453    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969455    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969459    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969462    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969464    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969467    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969470    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969472    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969475    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.981460 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969478    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969480    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969483    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969486    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969490    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969493    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969496    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969499    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969502    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969504    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969512    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969515    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969517    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969520    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969523    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969526    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969528    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969531    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969534    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.981954 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969537    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969540    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969542    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969546    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969550    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969553    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969556    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969564    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969566    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969569    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969572    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969574    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969577    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969579    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969582    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969584    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969587    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969589    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969591    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969594    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.982432 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969597    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969599    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969602    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969604    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969613    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969616    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969619    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969621    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969624    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969627    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969630    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969632    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969635    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969637    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969640    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969644    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969649    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969651    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969654    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.982913 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969656    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969659    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969662    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969664    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969667    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969669    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969672    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969674    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969677    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969679    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969682    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969684    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969687    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969689    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969692    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969694    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969697    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969699    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969707    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969710    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.983408 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969713    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969716    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969718    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969721    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969724    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969727    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969729    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.969732    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.975882    4214 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.975999    4214 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.976025    4214 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.976030    4214 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976088    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976093    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976096    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976099    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976102    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976104    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976107    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.983892 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976110    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976113    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976116    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976118    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976123    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976126    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976129    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976132    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976135    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976137    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976140    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976143    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976146    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976148    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976151    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976153    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976156    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976159    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976162    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976165    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.984383 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976168    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976172    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976174    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976177    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976180    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976182    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976185    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976187    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976190    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976193    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976196    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976198    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976201    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976203    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976206    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976208    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976211    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976214    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976216    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976220    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.984870 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976222    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976240    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976245    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976247    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976250    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976253    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976256    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976258    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976261    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976264    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976266    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976269    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976272    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976274    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976277    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976280    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976282    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976285    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976287    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.985354 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976290    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976292    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976295    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976297    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976300    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976302    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976305    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976307    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976310    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976313    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976316    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976319    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976321    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976324    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976327    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976330    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976332    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976335    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976338    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.985818 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976340    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976343    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976345    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976347    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976350    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976353    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976355    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976358    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976361    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976363    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976366    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976369    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976371    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976374    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976376    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976379    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976381    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976383    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976386    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.986346 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976388    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976393    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.976421    4214 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976552    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976557    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976560    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976562    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976565    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976568    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976571    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976574    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976577    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976579    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976582    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976585    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.986812 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976587    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976590    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976592    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976595    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976597    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976600    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976602    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976605    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976607    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976610    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976613    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976615    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976618    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976620    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976623    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976625    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976628    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976630    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976634    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976636    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.987193 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976640    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976642    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976645    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976647    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976650    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976652    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976656    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976658    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976661    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976663    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976666    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976668    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976671    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976673    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976676    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976678    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976681    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976683    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976686    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.987696 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976688    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976690    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976693    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976696    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976698    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976701    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976705    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976708    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976711    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976714    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976716    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976719    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976721    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976724    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976727    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976729    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976732    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976734    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976737    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976739    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.988145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976742    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976745    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976747    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976750    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976752    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976755    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976758    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976760    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976763    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976765    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976768    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976770    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976773    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976775    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976778    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976780    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976783    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976785    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976788    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976790    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.988656 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976793    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976795    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976799    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976803    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976807    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976810    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976813    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976815    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976818    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976821    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976823    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976826    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976828    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976832    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976834    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.989131 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.976856    4214 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976888    4214 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976891    4214 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976894    4214 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976896    4214 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976899    4214 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976901    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976904    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976907    4214 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976910    4214 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976914    4214 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976917    4214 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976920    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976923    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976926    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976928    4214 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976931    4214 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976933    4214 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:00.989499 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976936    4214 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976938    4214 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976942    4214 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976944    4214 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976947    4214 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976949    4214 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976952    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976954    4214 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976957    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976960    4214 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976962    4214 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976965    4214 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976967    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976971    4214 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976973    4214 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976976    4214 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976979    4214 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976981    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976984    4214 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976986    4214 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:00.989922 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976988    4214 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976991    4214 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976993    4214 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976996    4214 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.976998    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977001    4214 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977004    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977006    4214 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977009    4214 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977011    4214 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977014    4214 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977016    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977019    4214 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977021    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977024    4214 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977027    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977029    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977032    4214 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977034    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977037    4214 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:00.990405 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977040    4214 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977042    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977045    4214 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977047    4214 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977050    4214 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977052    4214 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977056    4214 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977059    4214 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977061    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977064    4214 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977066    4214 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977068    4214 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977071    4214 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977073    4214 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977076    4214 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977078    4214 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977081    4214 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977084    4214 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977087    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:00.990885 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977089    4214 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977092    4214 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977096    4214 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977098    4214 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977101    4214 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977104    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977106    4214 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977109    4214 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977111    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977114    4214 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977119    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977122    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977124    4214 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977127    4214 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977130    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977132    4214 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977135    4214 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977138    4214 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977140    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977143    4214 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:00.991370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977146    4214 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977148    4214 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977151    4214 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977153    4214 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977156    4214 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977159    4214 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977162    4214 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977164    4214 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977167    4214 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:00.977169    4214 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.977239    4214 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978189    4214 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978273    4214 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978456    4214 server.go:1014] "Starting client certificate rotation"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978593    4214 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978637    4214 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:21 +0000 UTC" deadline="2028-03-29 03:12:07.099764102 +0000 UTC"
Oct 30 02:41:00.991832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978671    4214 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12384h31m6.12109492s"
Oct 30 02:41:00.992399 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:00.978902    4214 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:01.001955 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.001940    4214 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:01.003680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.003663    4214 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:41:01.005632 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.005619    4214 log.go:25] "Validated CRI v1 image API"
Oct 30 02:41:01.006377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.006363    4214 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:41:01.008066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.008037    4214 fs.go:135] Filesystem UUIDs: map[21db0713-c6a8-4266-b88a-936b4e991d0d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cde6eb4c-f2ef-496a-8e8d-34d8610658ef:/dev/nvme0n1p3]
Oct 30 02:41:01.009341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.008065    4214 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e7a69e7612e387834aac81ee8d523e8aaabef16d664d3a20c7384686774a522/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e7a69e7612e387834aac81ee8d523e8aaabef16d664d3a20c7384686774a522/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf28a82a3f43d61879875907b55b442deb51a7f61a4a3e7f4af323cb3eccc6f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf28a82a3f43d61879875907b55b442deb51a7f61a4a3e7f4af323cb3eccc6f9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6/volumes/kubernetes.io~projected/kube-api-access-lkxcw:{mountpoint:/var/lib/kubelet/pods/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6/volumes/kubernetes.io~projected/kube-api-access-lkxcw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a968709-cf63-4ebc-a35e-aff8293d815c/volumes/kubernetes.io~projected/kube-api-access-vmhks:{mountpoint:/var/lib/kubelet/pods/0a968709-cf63-4ebc-a35e-aff8293d815c/volumes/kubernetes.io~projected/kube-api-access-vmhks major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f018c84-f211-4981-b33e-ad7e75ce95c9/volumes/kubernetes.io~projected/kube-api-access-rw6j5:{mountpoint:/var/lib/kubelet/pods/1f018c84-f211-4981-b33e-ad7e75ce95c9/volumes/kubernetes.io~projected/kube-api-access-rw6j5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3972a4c2-785f-4c36-967d-c3eb4bdd548e/volumes/kubernetes.io~projected/kube-api-access-xqc6l:{mountpoint:/var/lib/kubelet/pods/3972a4c2-785f-4c36-967d-c3eb4bdd548e/volumes/kubernetes.io~projected/kube-api-access-xqc6l major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~projected/kube-api-access-ltz9l:{mountpoint:/var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~projected/kube-api-access-ltz9l major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~projected/kube-api-access-6cw4l:{mountpoint:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~projected/kube-api-access-6cw4l major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6/volumes/kubernetes.io~projected/kube-api-access-8kzfw:{mountpoint:/var/lib/kubelet/pods/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6/volumes/kubernetes.io~projected/kube-api-access-8kzfw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~projected/kube-api-access-kv4ql:{mountpoint:/var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~projected/kube-api-access-kv4ql major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc384a9-fc05-4f98-8822-4e9dbaf34282/volumes/kubernetes.io~projected/kube-api-access-hp4kp:{mountpoint:/var/lib/kubelet/pods/7dc384a9-fc05-4f98-8822-4e9dbaf34282/volumes/kubernetes.io~projected/kube-api-access-hp4kp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81e5958e-50f9-4bde-948e-841d12e07b7d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/81e5958e-50f9-4bde-948e-841d12e07b7d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8578d008-b589-4708-b51f-f94fd8176e44/volumes/kubernetes.io~projected/kube-api-access-kv85j:{mountpoint:/var/lib/kubelet/pods/8578d008-b589-4708-b51f-f94fd8176e44/volumes/kubernetes.io~projected/kube-api-access-kv85j major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c811b314-2fff-49fd-abc8-0a8304fb94e6/volumes/kubernetes.io~projected/kube-api-access-bb7cw:{mountpoint:/var/lib/kubelet/pods/c811b314-2fff-49fd-abc8-0a8304fb94e6/volumes/kubernetes.io~projected/kube-api-access-bb7cw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7/volumes/kubernetes.io~projected/kube-api-access-62vq9:{mountpoint:/var/lib/kubelet/pods/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7/volumes/kubernetes.io~projected/kube-api-access-62vq9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~projected/kube-api-access-n7ncl:{mountpoint:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~projected/kube-api-access-n7ncl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f669d42f-1e2c-43c2-8172-202d70c39f0d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f669d42f-1e2c-43c2-8172-202d70c39f0d/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/81b1b4a873314748500f90397f4716cd3f226b455109947e1526ec46cb11cdfc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c7f81e04ddd5c213eaedaa93cedd26b5cb6afdd2bd658b902dc34b2fd7bd533a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/acb6d24711b61e881180a7f9e361950b0c0944c46dafa4aa0ef3e8802e7436a0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2351cf11fde936003320de54f5bdc0efd79a7553858a709c77a48b0fb7b48c3f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fb22c5e081f7add88df95654e1a088b070c3be8bca395c6b900a8a66fb5c030e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1f4376368f45463fb9c9c5006f12815efe00847f736ada1519f532fa69d7bc83/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d1b30c04ef8c31cfed5a05601e936810cb4953b61f25a545e50ab583ab8ad462/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e2c92a3b2556729610da90f76ae56afb8b91f8132ae369d92ae7da1e1a3bf383/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/bc2bbfb66102153739110953c323fda697586b6adb00f25c2970e85dbd347e8f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/58edfe32c00284742bfb1654615949079dffd2cc9cda7b2534aecebce6ad2372/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/ac4b3776595b235a02be3feb6fc029c2a4b4e9514969a5e084147ab8d0af45f2/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/aa07b57342b570147180f280245210e466b484a57b35fde1bac54f7619dda6a2/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6df6b9ae83c4eb880d49c19d1c0f9523a3f957762b0f0226ac6168a5e5ec77d1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c8a40dbc1c5ab8aa8a09f54d289f5fa28afd0cc84736eecf14daf240dd819383/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/5683ccc826a338f42760e178fc6198a121047965c7021bdddcacc193808ccf33/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/112bf99e63db7e24091813560e0e62bd8850f107e47447b8f12abe65af284828/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8b87d912ada16704f57d6e225a43b5cbb9ce6de6693c70f1bc4599d76586e963/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/09468f2b0465078926bf017b4d0aab0c3a02638ea88b7850a97ff385a1a7d6f1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6bd34193bcec601955585a527ef4ab5f950414264fc64f7bdec2215ea9b4fec7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dded8ad96e8daa664a8108060f82be3a8f05951460c5d5b08baa4fac3bc2a2c4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c3593de18231d8ed8f115795b8712889adeaaa2e05bda771005dd3304b69d7fe/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/5e4ee171c2f104c97597dab2b85b6a2c620950b26f056eb58807202041a28a46/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/479421399cab526031ddb6b4b480534687317976494a749c972a3bc33ab96464/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/810b41d9b791b46004f57fd30dbf8eed01d91c79c88fcde30abbbeb7cb436c42/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/be6570880d1bddd016fbfdfc828b0bc512bfbcaa9970344ea022c0de254dc47f/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/6da42289ac46d378f8588744a24a9a409720a2fb4b526040efbcc7f4e92b81dc/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/d28c7f95ee7a5a04a5bb5aed1fd014a4d60a29cad0a6f36456efd2149f0bf240/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/09e9ebe975d92e7e49c865ec4107eab60cc7e78ae740b95bbb3307068f27c3f6/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/91c06dfccd2bfa82c49ddc2a141a59f1e95a50be66ecd8eda91713f7f5883c2a/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/a50f38dd298b92aea336dd5496e0065f8b6e39fb0b48ae5317289629a7ca73f9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/8ae81c8307685e842cb9d9927cc659b1567372182b91e91eab49bb5b33fe85c5/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fb2dd4b8d9f63af3cab7ddad98e94c83369f08e7a5c514efbc776adb6e395f4a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/1f931afd61fd4ee6f0b5bfcef8a25e3d7eb3afef7932f51843495f82ea7f228a/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/aef8966649898ea6abbde4bc33ab078332ad7c1f3b55bb69c00e99715aa21ee4/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/06b61131a43f105e1d530d20369963df8c806df6fa659c36480f8526ff84e276/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9df860a77a24d0cfe3d74dd886c97d21c8ac79fbbe300f5f1689f2f91cdbe688/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/13a2577c53203286842da59e10bc06ea78586dfb8a285c28d694d68c38a7938e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cc9b7d1b7f090f62fec18e82d8fc3ff2de49b03b8a735cf71705a4e3920fbff3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cd9ff5a88d92e23a4489fecd042444a1e08e709d5f900cf53dbb4ef31a8a6a2b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/fe11ac481e4a34037251a6a00b0cdb32c5de6325337ca59b49a9a4c61f7d33e2/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 30 02:41:01.015545 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.015288    4214 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:01.014788235 +0000 UTC m=+0.088959020 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098659 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f6a47c8437412f13dca2f8183bf3 SystemUUID:ec22f6a4-7c84-3741-2f13-dca2f8183bf3 BootID:9dca3f60-45e6-4953-82a3-f607a1e996b7 Filesystems:[{Device:/var/lib/kubelet/pods/7dc384a9-fc05-4f98-8822-4e9dbaf34282/volumes/kubernetes.io~projected/kube-api-access-hp4kp DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/214c4de0ec5dc1c5a540850bb345463c059f4f3bc23eacfae1fa66f8fc467e51/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf28a82a3f43d61879875907b55b442deb51a7f61a4a3e7f4af323cb3eccc6f9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~projected/kube-api-access-kv4ql DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3972a4c2-785f-4c36-967d-c3eb4bdd548e/volumes/kubernetes.io~projected/kube-api-access-xqc6l DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/701387a9-97f7-42f7-b9f8-11a044f5a3c7/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81e5958e-50f9-4bde-948e-841d12e07b7d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7d81d56ca952b3bcc6aaba31f42d42dc190bcc66e6e5a9aa7019df3708ac350/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f669d42f-1e2c-43c2-8172-202d70c39f0d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~projected/kube-api-access-ltz9l DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e723255ad1f82461d494506e94f93a2f8c3ecd790e1e39f2dc046056d52a2586/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e1acad5-796c-4ea8-b791-ce810c9438dd/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6/volumes/kubernetes.io~projected/kube-api-access-lkxcw DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e7a69e7612e387834aac81ee8d523e8aaabef16d664d3a20c7384686774a522/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47197dda1d317c7cd867cf42e40bf108063af716e0b011e0e4c94efcbefd4e28/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34bd87bac6e76777f15a031f941bb311671408dc9d1fb5582cb75dedd8d7aa77/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b334078141fd48d68362991c79c519a94b587115214fed05fa880de7b362411c/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c811b314-2fff-49fd-abc8-0a8304fb94e6/volumes/kubernetes.io~projected/kube-api-access-bb7cw DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49054a0489cccddcfb0136a2480f317858fd901c16c853b59f7b225dea1dbcfa/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6/volumes/kubernetes.io~projected/kube-api-access-8kzfw DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volumes/kubernetes.io~projected/kube-api-access-n7ncl DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0a968709-cf63-4ebc-a35e-aff8293d815c/volumes/kubernetes.io~projected/kube-api-access-vmhks DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5af13450-f1f2-4b72-a329-3dd739b9a10e/volumes/kubernetes.io~projected/kube-api-access-6cw4l DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1ccb827cda662ce2ff707bbd7143a30eccf3f1af069ac6077acde597584d07f/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f05bec463633c51cbfd1287270a1eb9b6ad7947b23a4e918efd4791b4917cfa5/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f018c84-f211-4981-b33e-ad7e75ce95c9/volumes/kubernetes.io~projected/kube-api-access-rw6j5 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7/volumes/kubernetes.io~projected/kube-api-access-62vq9 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f880ceeb7b370eac8194918be450b4e48617b9063fe64fed7ca38854653460a3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8578d008-b589-4708-b51f-f94fd8176e44/volumes/kubernetes.io~projected/kube-api-access-kv85j DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f1de0f3c-4516-4210-afaa-3657c54257af/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:b334078141fd48d MacAddress:26:ac:a9:a2:dc:f2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d6:1f:7b:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:2a:9e:4f:1f:b0 Speed:0 Mtu:8901} {Name:e723255ad1f8246 MacAddress:76:21:ea:9f:cb:22 Speed:10000 Mtu:8901} {Name:e7d81d56ca952b3 MacAddress:8a:f7:39:dc:33:6e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d6:1f:7b:f1 Speed:0 Mtu:9001} {Name:f05bec463633c51 MacAddress:6a:d7:7f:1f:43:db Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:32:6c:36:a4:a2:1d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:92:0f:e2:76:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:41:01.015722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.015549    4214 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:41:01.015722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.015652    4214 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:41:01.016001 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.015968    4214 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:41:01.016146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016002    4214 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:41:01.016222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016149    4214 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:41:01.016222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016161    4214 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:41:01.016222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016175    4214 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:01.016222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016202    4214 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:01.016435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016298    4214 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:41:01.016533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016520    4214 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:41:01.016648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016638    4214 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:41:01.016700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016682    4214 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:41:01.016700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016696    4214 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:41:01.016793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016711    4214 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:41:01.016793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016725    4214 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:41:01.016793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.016736    4214 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:41:01.017381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.017364    4214 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:01.017464 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.017384    4214 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:01.017840 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.017820    4214 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:41:01.020156 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020140    4214 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:01.020156 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020154    4214 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:01.020279 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020269    4214 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:41:01.020467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020456    4214 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:41:01.020499 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020485    4214 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:41:01.020574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020566    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:41:01.020603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020580    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:41:01.020603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020587    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:41:01.020603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020593    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:41:01.020603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020598    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:41:01.020603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020604    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020610    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020616    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020622    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020627    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020643    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020653    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020685    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:41:01.020741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.020691    4214 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:41:01.023965 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.023949    4214 server.go:1263] "Started kubelet"
Oct 30 02:41:01.024053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.024025    4214 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:41:01.024604 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.024055    4214 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:41:01.024687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.024623    4214 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:41:01.024945 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.024898    4214 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:41:01.025137 ip-10-0-136-171 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:41:01.025942 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.025928    4214 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:41:01.027769 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.027750    4214 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:41:01.028735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.028718    4214 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:01.029567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.029529    4214 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:41:01.029821 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.029791    4214 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-171.ec2.internal\" not found"
Oct 30 02:41:01.030665 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.029912    4214 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:41:01.030665 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.029984    4214 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:41:01.031811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031788    4214 factory.go:55] Registering systemd factory
Oct 30 02:41:01.031811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031807    4214 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031804    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701387a9-97f7-42f7-b9f8-11a044f5a3c7" volumeName="kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031834    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7" volumeName="kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031824    4214 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:23 +0000 UTC" deadline="2028-05-14 02:27:46.454901878 +0000 UTC"
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031857    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a968709-cf63-4ebc-a35e-aff8293d815c" volumeName="kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031861    4214 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h46m45.423044436s"
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031873    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af13450-f1f2-4b72-a329-3dd739b9a10e" volumeName="kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031889    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" volumeName="kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031902    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701387a9-97f7-42f7-b9f8-11a044f5a3c7" volumeName="kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031915    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701387a9-97f7-42f7-b9f8-11a044f5a3c7" volumeName="kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031937    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1de0f3c-4516-4210-afaa-3657c54257af" volumeName="kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:41:01.031950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031952    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1de0f3c-4516-4210-afaa-3657c54257af" volumeName="kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031969    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f018c84-f211-4981-b33e-ad7e75ce95c9" volumeName="kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031981    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3972a4c2-785f-4c36-967d-c3eb4bdd548e" volumeName="kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.031994    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e1acad5-796c-4ea8-b791-ce810c9438dd" volumeName="kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032011    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e1acad5-796c-4ea8-b791-ce810c9438dd" volumeName="kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032023    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f669d42f-1e2c-43c2-8172-202d70c39f0d" volumeName="kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032045    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc384a9-fc05-4f98-8822-4e9dbaf34282" volumeName="kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032057    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1de0f3c-4516-4210-afaa-3657c54257af" volumeName="kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032075    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8578d008-b589-4708-b51f-f94fd8176e44" volumeName="kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032088    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f669d42f-1e2c-43c2-8172-202d70c39f0d" volumeName="kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032106    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f018c84-f211-4981-b33e-ad7e75ce95c9" volumeName="kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032118    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af13450-f1f2-4b72-a329-3dd739b9a10e" volumeName="kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032130    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1de0f3c-4516-4210-afaa-3657c54257af" volumeName="kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032155    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1de0f3c-4516-4210-afaa-3657c54257af" volumeName="kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032169    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8578d008-b589-4708-b51f-f94fd8176e44" volumeName="kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032187    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c811b314-2fff-49fd-abc8-0a8304fb94e6" volumeName="kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032200    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3972a4c2-785f-4c36-967d-c3eb4bdd548e" volumeName="kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032213    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3972a4c2-785f-4c36-967d-c3eb4bdd548e" volumeName="kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l" seLinuxMountContext=""
Oct 30 02:41:01.032461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032458    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" volumeName="kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032476    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc384a9-fc05-4f98-8822-4e9dbaf34282" volumeName="kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032503    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6" volumeName="kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032517    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a968709-cf63-4ebc-a35e-aff8293d815c" volumeName="kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032534    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" volumeName="kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032547    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701387a9-97f7-42f7-b9f8-11a044f5a3c7" volumeName="kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032563    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81e5958e-50f9-4bde-948e-841d12e07b7d" volumeName="kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.032576    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f018c84-f211-4981-b33e-ad7e75ce95c9" volumeName="kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.033093    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5af13450-f1f2-4b72-a329-3dd739b9a10e" volumeName="kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.033126    4214 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" volumeName="kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.033141    4214 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:41:01.033144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.033150    4214 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:41:01.034413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034394    4214 factory.go:153] Registering CRI-O factory
Oct 30 02:41:01.034500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034415    4214 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:41:01.034500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034467    4214 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:41:01.034500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034488    4214 factory.go:103] Registering Raw factory
Oct 30 02:41:01.034637 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034504    4214 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:41:01.034908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.034891    4214 manager.go:319] Starting recovery of all containers
Oct 30 02:41:01.055106 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.055067    4214 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:41:01.058336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.058314    4214 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:41:01.058336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.058337    4214 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:41:01.058500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.058356    4214 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:41:01.058500 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.058392    4214 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:41:01.066023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066000    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="183fa1c0a9046968981c0ec85a74ae1aa0b4fc90d5747edb5f3a231ada06e9e1" exitCode=0
Oct 30 02:41:01.066023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066018    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="a6c813a7091bb59bfe1677c132115143356f50a50b64a7dcf7249981f8a81c0f" exitCode=0
Oct 30 02:41:01.066023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066025    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="d59710682bbc9f357987584a8c7eb0d936d1fa5b89d95e97e158c61539e8dbb4" exitCode=0
Oct 30 02:41:01.066267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066031    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="fb4bbbdb05aa5c1fcefc2b0bbda7173ccb72cbf7e08413b6016bc149870a0193" exitCode=0
Oct 30 02:41:01.066267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066036    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="59e9c9d444761cd997fd20e36b02e0876bb9f3aa1b4bef9b9c7a0b15ef3d4710" exitCode=0
Oct 30 02:41:01.066267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.066041    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6" containerID="00c488e7607aee04e96b5c5332487261c8523f76003a2e3154f424a2073ce01a" exitCode=0
Oct 30 02:41:01.077974 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.077957    4214 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:41:01.078753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.078725    4214 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:01.079012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.078992    4214 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:01.085237 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.085198    4214 generic.go:356] "Generic (PLEG): container finished" podID="46b9d2d1e9934b263ff5b61e7cf24e45" containerID="33a7f6d1bbbe873e802ff4e278ab413f7899099cc1d655d8721f12f639d773a1" exitCode=0
Oct 30 02:41:01.092221 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.092201    4214 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:01.092563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.092543    4214 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:01.102136 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.102118    4214 manager.go:324] Recovery completed
Oct 30 02:41:01.115063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115014    4214 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:41:01.115063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115026    4214 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:41:01.115063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115040    4214 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:41:01.115196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115147    4214 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:41:01.115196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115157    4214 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:41:01.115196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115189    4214 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:41:01.115196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115194    4214 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:41:01.115328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115199    4214 policy_none.go:50] "Start"
Oct 30 02:41:01.115328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115207    4214 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:41:01.115328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115215    4214 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:41:01.115410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115354    4214 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:41:01.115410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115361    4214 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:41:01.115410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.115366    4214 policy_none.go:44] "Start"
Oct 30 02:41:01.119358 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.119336    4214 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:41:01.121835 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.121824    4214 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:41:01.122282 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.121845    4214 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:41:01.122342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122284    4214 server.go:87] "Starting device plugin registration server"
Oct 30 02:41:01.122518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122509    4214 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:41:01.122554 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122519    4214 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:41:01.123173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122622    4214 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:41:01.123173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122711    4214 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:41:01.123173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122751    4214 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:41:01.123173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122758    4214 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:41:01.123173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.122809    4214 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:01.122738064Z","UUID":"371d101d-8268-4263-b867-d819a5a75dfb","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:41:01.124030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.124006    4214 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:41:01.124110 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.124030    4214 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:41:01.124110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.124039    4214 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:41:01.158633 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.158590    4214 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"]
Oct 30 02:41:01.159110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159072    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerStarted","Data":"14ac0da8c88a3340bbe907f2979d5a5bb6cedf0faa1ad9dac4cfe07aae5b253c"}
Oct 30 02:41:01.159110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159107    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerDied","Data":"33a7f6d1bbbe873e802ff4e278ab413f7899099cc1d655d8721f12f639d773a1"}
Oct 30 02:41:01.159266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159120    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal" event={"ID":"46b9d2d1e9934b263ff5b61e7cf24e45","Type":"ContainerStarted","Data":"3fbbefe0b9437f233b2051ebe1798440aadc4fbae2ef160be016e9a1f517f13a"}
Oct 30 02:41:01.159266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159142    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal" event={"ID":"05848580451918f708548c7de17eb55a","Type":"ContainerStarted","Data":"44deb679dcbf480a009c443456dbd28add4c7b56a49334332f857e671ab37ad4"}
Oct 30 02:41:01.159266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159145    4214 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.159266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159153    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal" event={"ID":"05848580451918f708548c7de17eb55a","Type":"ContainerStarted","Data":"6a36505fad97dfd864b7cde6a717aa4681d5a22fc12d08c0186124768ee004b2"}
Oct 30 02:41:01.159413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.159318    4214 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.182266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.182249    4214 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:01.182323 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.182283    4214 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.182672 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.182659    4214 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:01.182714 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:01.182693    4214 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.223381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.223362    4214 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:01.224612 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.224599    4214 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:01.224686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.224625    4214 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:01.224686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.224634    4214 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-171.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:01.224686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.224662    4214 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.261050 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.261024    4214 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.261153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.261090    4214 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.278819 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.278791    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.278953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.278823    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.278953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.278850    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.378982    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.379021    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.379039    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.379073    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.379085    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05848580451918f708548c7de17eb55a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" (UID: \"05848580451918f708548c7de17eb55a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:01.379288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:01.379093    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46b9d2d1e9934b263ff5b61e7cf24e45-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" (UID: \"46b9d2d1e9934b263ff5b61e7cf24e45\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:02.018041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.018009    4214 apiserver.go:52] "Watching apiserver"
Oct 30 02:41:02.095175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.095145    4214 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:02.095334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.095295    4214 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:02.097675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.097656    4214 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:41:02.098075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.098056    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khc7g","openshift-network-diagnostics/network-check-target-8xf7s","openshift-dns/node-resolver-cqnzm","kube-system/global-pull-secret-syncer-d6zml","openshift-multus/multus-cjx5c","openshift-multus/multus-additional-cni-plugins-w97ss","openshift-ovn-kubernetes/ovnkube-node-2cw8z","kube-system/konnectivity-agent-5v56d","openshift-multus/network-metrics-daemon-pkkvd","openshift-network-operator/iptables-alerter-8lxnt","kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572","openshift-image-registry/node-ca-xtkw9","openshift-cluster-node-tuning-operator/tuned-bqdtf","openshift-ingress-canary/ingress-canary-7k44l","openshift-insights/insights-runtime-extractor-c4grk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"]
Oct 30 02:41:02.098497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.098480    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.105976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.105957    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.126202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.126183    4214 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:02.126307 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:02.126257    4214 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-171.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:02.130148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.130134    4214 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:02.130210 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:02.130166    4214 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-171.ec2.internal"
Oct 30 02:41:02.130744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.130729    4214 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:41:02.130791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.130747    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:41:02.131040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131028    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:41:02.131300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131286    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:41:02.131415 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131392    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:41:02.131525 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131483    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.131582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131524    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.131942 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.131931    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:41:02.132155 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.132144    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:41:02.132206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.132192    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:41:02.132206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.132199    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:41:02.132838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.132826    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:41:02.133497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.133485    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:41:02.133555 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.133488    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:41:02.133992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.133980    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.134058 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134009    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:41:02.134194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134182    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.134379 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134365    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:41:02.134483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134460    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:41:02.134580 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134567    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:41:02.134729 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134717    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:41:02.134873 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134862    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.134942 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.134894    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:41:02.135026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.135013    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.135110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.135100    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.135152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.135132    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.135293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.135282    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:41:02.135448 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.135437    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:41:02.137882 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.137870    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:41:02.146949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.146932    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xn2tj\""
Oct 30 02:41:02.146995 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.146988    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:41:02.147714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147652    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.147714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147699    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:41:02.147959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147852    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.147959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147854    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:41:02.147959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147882    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:41:02.147959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147903    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.147959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.147972    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148025    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148030    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148032    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148076    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148103    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148102    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148151    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd75s\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148152    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148269    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148322    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148455    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148467    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148469    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148468    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148529    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.148709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.148584    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:41:02.149617 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.149604    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:41:02.149662 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.149639    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdxrj\""
Oct 30 02:41:02.181561 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181541    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.181644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181563    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.181644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181582    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.181644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181598    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vq9\" (UniqueName: \"kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.181644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181614    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4ql\" (UniqueName: \"kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.181814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181655    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.181814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181724    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cw4l\" (UniqueName: \"kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.181814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181770    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.181945 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181825    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.181994 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.181970    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.182053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182027    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhks\" (UniqueName: \"kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.182107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182055    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-cni-binary-copy\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.182107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182040    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.182107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182072    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182156    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:41:02.182274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182198    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.182274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182223    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.182370 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182284    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.182370 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182316    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.182422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182373    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-env-overrides\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182385    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182426    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ncl\" (UniqueName: \"kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e1acad5-796c-4ea8-b791-ce810c9438dd-cert\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:41:02.182477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182472    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.182565 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182551    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.182594 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182582    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.182628 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182610    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.182668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182655    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182654    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3972a4c2-785f-4c36-967d-c3eb4bdd548e-data-volume\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.182721 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182693    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.182748 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182722    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:41:02.182840 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182765    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv85j\" (UniqueName: \"kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.182923 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182859    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.182961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182908    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.182961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-script-lib\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.183028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182958    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.183028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.182990    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.183028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183022    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.183164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183057    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzfw\" (UniqueName: \"kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.183164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183104    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxcw\" (UniqueName: \"kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.183164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183136    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.183330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183252    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.183330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183285    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.183431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183394    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.183431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183427    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.183534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183463    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.183534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.183634 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183572    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.183634 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183584    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:41:02.183634 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183620    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.183783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183640    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.183783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183693    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.183783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183728    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.183783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183758    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4kp\" (UniqueName: \"kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.183976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183782    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81e5958e-50f9-4bde-948e-841d12e07b7d-original-pull-secret\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.183976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183786    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.183976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.183976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183936    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f669d42f-1e2c-43c2-8172-202d70c39f0d-agent-certs\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:41:02.183976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183962    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1de0f3c-4516-4210-afaa-3657c54257af-ovn-node-metrics-cert\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.184242 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.183977    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.184242 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184116    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/701387a9-97f7-42f7-b9f8-11a044f5a3c7-metrics-tls\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.184456 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184043    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.184504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184464    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.184504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184483    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.184504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184488    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-tuned\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.184504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184498    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.184504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184100    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af13450-f1f2-4b72-a329-3dd739b9a10e-tmp\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.184663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184527    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.184663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184622    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.184663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184643    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.184832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184663    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc6l\" (UniqueName: \"kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.184832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184672    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-daemon-config\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.184832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184694    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.184832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184713    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184837    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6j5\" (UniqueName: \"kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184853    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184867    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184882    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184895    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184916    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184939    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.184972    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz9l\" (UniqueName: \"kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185027    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185053    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f669d42f-1e2c-43c2-8172-202d70c39f0d-konnectivity-ca\") pod \"konnectivity-agent-5v56d\" (UID: \"f669d42f-1e2c-43c2-8172-202d70c39f0d\") " pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185059    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185166    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1de0f3c-4516-4210-afaa-3657c54257af-ovnkube-config\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185179    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185251    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185282    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185309    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185329    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185343    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185355    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185388    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185423    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185445    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185446    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8578d008-b589-4708-b51f-f94fd8176e44-tmp-dir\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185468    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185497    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185513    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185527    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185540    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185556    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185570    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185607    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185628    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185643    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185664    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185717    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185782    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/701387a9-97f7-42f7-b9f8-11a044f5a3c7-config-volume\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.185807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185788    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.186335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185818    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701387a9-97f7-42f7-b9f8-11a044f5a3c7-tmp-dir\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.186335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185856    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.186335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.185987    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc384a9-fc05-4f98-8822-4e9dbaf34282-iptables-alerter-script\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.186335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.186306    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a968709-cf63-4ebc-a35e-aff8293d815c-serviceca\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.200769 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.200751    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4ql\" (UniqueName: \"kubernetes.io/projected/701387a9-97f7-42f7-b9f8-11a044f5a3c7-kube-api-access-kv4ql\") pod \"dns-default-khc7g\" (UID: \"701387a9-97f7-42f7-b9f8-11a044f5a3c7\") " pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:02.201597 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.201582    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vq9\" (UniqueName: \"kubernetes.io/projected/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kube-api-access-62vq9\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.202199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.202184    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhks\" (UniqueName: \"kubernetes.io/projected/0a968709-cf63-4ebc-a35e-aff8293d815c-kube-api-access-vmhks\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.203167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.203147    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4kp\" (UniqueName: \"kubernetes.io/projected/7dc384a9-fc05-4f98-8822-4e9dbaf34282-kube-api-access-hp4kp\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.203272 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.203203    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzfw\" (UniqueName: \"kubernetes.io/projected/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-kube-api-access-8kzfw\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.203341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.203262    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cw4l\" (UniqueName: \"kubernetes.io/projected/5af13450-f1f2-4b72-a329-3dd739b9a10e-kube-api-access-6cw4l\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.203839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.203819    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxcw\" (UniqueName: \"kubernetes.io/projected/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-kube-api-access-lkxcw\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.203895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.203864    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ncl\" (UniqueName: \"kubernetes.io/projected/f1de0f3c-4516-4210-afaa-3657c54257af-kube-api-access-n7ncl\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.204023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.204008    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7cw\" (UniqueName: \"kubernetes.io/projected/c811b314-2fff-49fd-abc8-0a8304fb94e6-kube-api-access-bb7cw\") pod \"network-check-target-8xf7s\" (UID: \"c811b314-2fff-49fd-abc8-0a8304fb94e6\") " pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:41:02.204081 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.204064    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv85j\" (UniqueName: \"kubernetes.io/projected/8578d008-b589-4708-b51f-f94fd8176e44-kube-api-access-kv85j\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.204986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.204972    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc6l\" (UniqueName: \"kubernetes.io/projected/3972a4c2-785f-4c36-967d-c3eb4bdd548e-kube-api-access-xqc6l\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.206437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.206422    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6j5\" (UniqueName: \"kubernetes.io/projected/1f018c84-f211-4981-b33e-ad7e75ce95c9-kube-api-access-rw6j5\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.206496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.206423    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz9l\" (UniqueName: \"kubernetes.io/projected/3e1acad5-796c-4ea8-b791-ce810c9438dd-kube-api-access-ltz9l\") pod \"ingress-canary-7k44l\" (UID: \"3e1acad5-796c-4ea8-b791-ce810c9438dd\") " pod="openshift-ingress-canary/ingress-canary-7k44l"
Oct 30 02:41:02.286365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286302    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286335    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286349    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286366    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286382    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286400    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286420    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286411    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-slash\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286421    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-kubelet\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286443    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286427    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286462    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-socket-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286465    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286494    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-sys\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286480    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysconfig\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286503    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-bin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286501    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286528    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-node-log\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.286544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286541    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286557    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-netns\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286575    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-cnibin\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286595    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286623    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-lib-modules\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286657    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286682    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286661    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-registration-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286922    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-host\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.286969    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287015    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287042    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287075    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287012    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-conf-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287156    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-systemd\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287268    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287314    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287344    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287374    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287406    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287431    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287477    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.287687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287509    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.288090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.287976    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.288090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288008    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.288090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288032    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.288090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288059    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.288090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288086    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288113    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288141    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288168    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288200    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288244    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288272    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288297    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288324    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288352    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288380    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288408    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288447    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288478    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288508    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288540    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288567    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.288611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288592    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288622    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288650    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288681    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288711    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288740    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288765    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288792    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288823    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288917    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-hostroot\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.288970    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-netns\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289012    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-system-cni-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289124    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-modprobe-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289168    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-etc-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289222    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-kubelet-config\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289319    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-kubernetes\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289364    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-etc-kubernetes\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289424    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w97ss\" (UID: \"5e0ede86-75f7-4c52-ab1d-d5d6cb8747a6\") " pod="openshift-multus/multus-additional-cni-plugins-w97ss"
Oct 30 02:41:02.289493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289468    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a968709-cf63-4ebc-a35e-aff8293d815c-host\") pod \"node-ca-xtkw9\" (UID: \"0a968709-cf63-4ebc-a35e-aff8293d815c\") " pod="openshift-image-registry/node-ca-xtkw9"
Oct 30 02:41:02.289787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289519    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.289787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289570    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.289787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289684    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-conf\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.289787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289731    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-systemd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.289787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289784    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81e5958e-50f9-4bde-948e-841d12e07b7d-dbus\") pod \"global-pull-secret-syncer-d6zml\" (UID: \"81e5958e-50f9-4bde-948e-841d12e07b7d\") " pod="kube-system/global-pull-secret-syncer-d6zml"
Oct 30 02:41:02.289987 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289833    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-cnibin\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289987 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289875    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-k8s-cni-cncf-io\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.289987 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289926    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-bin\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.289973    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-device-dir\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.290169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290115    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-log-socket\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290075    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc384a9-fc05-4f98-8822-4e9dbaf34282-host-slash\") pod \"iptables-alerter-8lxnt\" (UID: \"7dc384a9-fc05-4f98-8822-4e9dbaf34282\") " pod="openshift-network-operator/iptables-alerter-8lxnt"
Oct 30 02:41:02.290169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290144    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-kubelet\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.290169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290157    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-var-lib-cni-multus\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.290169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290083    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-run\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290075    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7-sys-fs\") pod \"aws-ebs-csi-driver-node-mp572\" (UID: \"eedbd56c-4ed3-4ba9-a6c6-674d0d0973f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mp572"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290173    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8578d008-b589-4708-b51f-f94fd8176e44-hosts-file\") pod \"node-resolver-cqnzm\" (UID: \"8578d008-b589-4708-b51f-f94fd8176e44\") " pod="openshift-dns/node-resolver-cqnzm"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290180    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-etc-sysctl-d\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290119    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3972a4c2-785f-4c36-967d-c3eb4bdd548e-crio-socket\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290050    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-run-ovn\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290261    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-systemd-units\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290085    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-var-lib-openvswitch\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290257    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5af13450-f1f2-4b72-a329-3dd739b9a10e-var-lib-kubelet\") pod \"tuned-bqdtf\" (UID: \"5af13450-f1f2-4b72-a329-3dd739b9a10e\") " pod="openshift-cluster-node-tuning-operator/tuned-bqdtf"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290318    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-host-run-multus-certs\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.290392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290323    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290735    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-cni-netd\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.290813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290768    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-system-cni-dir\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.290813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290786    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f018c84-f211-4981-b33e-ad7e75ce95c9-multus-socket-dir-parent\") pod \"multus-cjx5c\" (UID: \"1f018c84-f211-4981-b33e-ad7e75ce95c9\") " pod="openshift-multus/multus-cjx5c"
Oct 30 02:41:02.290813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.290792    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1de0f3c-4516-4210-afaa-3657c54257af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2cw8z\" (UID: \"f1de0f3c-4516-4210-afaa-3657c54257af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:02.291197 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.291171    4214 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:41:02.294314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.294295    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3972a4c2-785f-4c36-967d-c3eb4bdd548e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c4grk\" (UID: \"3972a4c2-785f-4c36-967d-c3eb4bdd548e\") " pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.294402 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.294297    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6-metrics-certs\") pod \"network-metrics-daemon-pkkvd\" (UID: \"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6\") " pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.398894 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.398871    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pkkvd"
Oct 30 02:41:02.406734 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.406714    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c4grk"
Oct 30 02:41:02.739002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.738795    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pkkvd"]
Oct 30 02:41:02.743347 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:02.743303    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a21cf8c_c9f5_47ce_b6cd_e5a6b8e7c8a6.slice/crio-b7c2b84133cd4a2cbda4f6ec5e52a118369bfa5999277e8a4395927c50f552ea WatchSource:0}: Error finding container b7c2b84133cd4a2cbda4f6ec5e52a118369bfa5999277e8a4395927c50f552ea: Status 404 returned error can't find the container with id b7c2b84133cd4a2cbda4f6ec5e52a118369bfa5999277e8a4395927c50f552ea
Oct 30 02:41:02.745105 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.745091    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:41:02.766538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.766516    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c4grk"]
Oct 30 02:41:02.771060 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:02.771032    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3972a4c2_785f_4c36_967d_c3eb4bdd548e.slice/crio-ecae7e23aa486784f49ea46415d17ffaeae2ccf5edfbec70beb33fdf27c67e06 WatchSource:0}: Error finding container ecae7e23aa486784f49ea46415d17ffaeae2ccf5edfbec70beb33fdf27c67e06: Status 404 returned error can't find the container with id ecae7e23aa486784f49ea46415d17ffaeae2ccf5edfbec70beb33fdf27c67e06
Oct 30 02:41:02.946604 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:02.946435    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:03.099182 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:03.099094    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c4grk" event={"ID":"3972a4c2-785f-4c36-967d-c3eb4bdd548e","Type":"ContainerStarted","Data":"cbf742cef6d9ad5bbc58a9f8a9b08eea08e1871a9b12723526ee9ed5c942b1d4"}
Oct 30 02:41:03.099182 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:03.099149    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c4grk" event={"ID":"3972a4c2-785f-4c36-967d-c3eb4bdd548e","Type":"ContainerStarted","Data":"ecae7e23aa486784f49ea46415d17ffaeae2ccf5edfbec70beb33fdf27c67e06"}
Oct 30 02:41:03.100775 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:03.100751    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pkkvd" event={"ID":"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6","Type":"ContainerStarted","Data":"b7c2b84133cd4a2cbda4f6ec5e52a118369bfa5999277e8a4395927c50f552ea"}
Oct 30 02:41:03.109159 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:03.109140    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:05.109926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:05.109435    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pkkvd" event={"ID":"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6","Type":"ContainerStarted","Data":"77b762ef1a9e3bc1e6d120e6e649c42b5471cad7b6d2590fa06a09d1d080d867"}
Oct 30 02:41:05.109926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:05.109505    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pkkvd" event={"ID":"0a21cf8c-c9f5-47ce-b6cd-e5a6b8e7c8a6","Type":"ContainerStarted","Data":"32061780d2deaf514f0157d6f7e790eb1ea016085097decb2554a6d8d04b5e0b"}
Oct 30 02:41:05.111594 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:05.111566    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c4grk" event={"ID":"3972a4c2-785f-4c36-967d-c3eb4bdd548e","Type":"ContainerStarted","Data":"b3fb7b3c9f3eb8d73219cfc9f2be438877acc5608977007e9a844e621b923048"}
Oct 30 02:41:06.103663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:06.103636    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khc7g"
Oct 30 02:41:06.116411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:06.116382    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c4grk" event={"ID":"3972a4c2-785f-4c36-967d-c3eb4bdd548e","Type":"ContainerStarted","Data":"d25177f6a127faeb5620c57b4e975f7dd33f8981649eacfe67b147e2bb39deb8"}
Oct 30 02:41:06.980023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:06.979987    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:06.980198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:06.980173    4214 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:09.161757 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:09.161686    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:41:09.476177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:09.476077    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:41:11.966149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:11.966119    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2cw8z"
Oct 30 02:41:12.509106 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.509076    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5677cfbdd5-87kq2"]
Oct 30 02:41:12.510952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.510936    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.553818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.553789    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:41:12.554103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.554063    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:41:12.554204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.554109    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fd755\""
Oct 30 02:41:12.554204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.554147    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:41:12.554204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.554197    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:41:12.554793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.554776    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5677cfbdd5-87kq2"]
Oct 30 02:41:12.644842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.644813    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-registry-certificates\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.644954 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.644848    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-image-registry-private-configuration\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.644954 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.644881    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-registry-tls\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.645038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.644960    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnsh\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-kube-api-access-ncnsh\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.645077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.645060    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-bound-sa-token\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.645114 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.645081    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9696533-3759-4de0-9d3c-5df1d9416911-ca-trust-extracted\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.645114 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.645105    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-trusted-ca\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.645183 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.645121    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-installation-pull-secrets\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745416    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9696533-3759-4de0-9d3c-5df1d9416911-ca-trust-extracted\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745456    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-trusted-ca\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745473    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-installation-pull-secrets\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745593    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-registry-certificates\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745658 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745630    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-image-registry-private-configuration\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745706 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745663    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-registry-tls\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745706 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745695    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnsh\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-kube-api-access-ncnsh\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745733    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-bound-sa-token\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.745950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.745927    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9696533-3759-4de0-9d3c-5df1d9416911-ca-trust-extracted\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.746408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.746377    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-registry-certificates\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.746408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.746401    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9696533-3759-4de0-9d3c-5df1d9416911-trusted-ca\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.749563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.749530    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-installation-pull-secrets\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.749563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.749561    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-registry-tls\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.749904 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.749884    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9696533-3759-4de0-9d3c-5df1d9416911-image-registry-private-configuration\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.765035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.764989    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-bound-sa-token\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.765035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.764992    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnsh\" (UniqueName: \"kubernetes.io/projected/f9696533-3759-4de0-9d3c-5df1d9416911-kube-api-access-ncnsh\") pod \"image-registry-5677cfbdd5-87kq2\" (UID: \"f9696533-3759-4de0-9d3c-5df1d9416911\") " pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:12.821712 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:12.821684    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:13.143095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:13.143068    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5677cfbdd5-87kq2"]
Oct 30 02:41:13.146833 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:13.146808    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9696533_3759_4de0_9d3c_5df1d9416911.slice/crio-bc6ba5039fc67e137b839f7cb70ac16137465c62fdc976532df36c92a94ff18b WatchSource:0}: Error finding container bc6ba5039fc67e137b839f7cb70ac16137465c62fdc976532df36c92a94ff18b: Status 404 returned error can't find the container with id bc6ba5039fc67e137b839f7cb70ac16137465c62fdc976532df36c92a94ff18b
Oct 30 02:41:14.140299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:14.140258    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2" event={"ID":"f9696533-3759-4de0-9d3c-5df1d9416911","Type":"ContainerStarted","Data":"fca1cc2357df3e73b88b066bb17b624032702423d45e2e9428cff31225c09236"}
Oct 30 02:41:14.140462 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:14.140351    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2" event={"ID":"f9696533-3759-4de0-9d3c-5df1d9416911","Type":"ContainerStarted","Data":"bc6ba5039fc67e137b839f7cb70ac16137465c62fdc976532df36c92a94ff18b"}
Oct 30 02:41:14.140462 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:14.140400    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:14.185448 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:14.185406    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2" podStartSLOduration=2.185395101 podStartE2EDuration="2.185395101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:14.184682484 +0000 UTC m=+13.258853280" watchObservedRunningTime="2026-10-30 02:41:14.185395101 +0000 UTC m=+13.259565891"
Oct 30 02:41:15.602802 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.602773    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"]
Oct 30 02:41:15.605905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.605886    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.635744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.635715    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 02:41:15.635744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.635726    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gc994\""
Oct 30 02:41:15.635921 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.635748    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 02:41:15.638582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.638563    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:41:15.638677 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.638592    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:41:15.638739 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.638699    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:41:15.649626 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.649604    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"]
Oct 30 02:41:15.765241 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.765197    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6626491-0f7e-456e-b1ff-e8f93e2325cd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.765371 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.765247    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.765371 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.765266    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4sz\" (UniqueName: \"kubernetes.io/projected/d6626491-0f7e-456e-b1ff-e8f93e2325cd-kube-api-access-2m4sz\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.765444 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.765370    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.866016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.865942    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.866016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.865994    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6626491-0f7e-456e-b1ff-e8f93e2325cd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.866016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.866015    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.866265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.866033    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4sz\" (UniqueName: \"kubernetes.io/projected/d6626491-0f7e-456e-b1ff-e8f93e2325cd-kube-api-access-2m4sz\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.866734 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.866709    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6626491-0f7e-456e-b1ff-e8f93e2325cd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.868379 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.868358    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.868461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.868414    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6626491-0f7e-456e-b1ff-e8f93e2325cd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.894627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.894609    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4sz\" (UniqueName: \"kubernetes.io/projected/d6626491-0f7e-456e-b1ff-e8f93e2325cd-kube-api-access-2m4sz\") pod \"prometheus-operator-9cd89b74c-tdffn\" (UID: \"d6626491-0f7e-456e-b1ff-e8f93e2325cd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:15.916583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:15.916564    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"
Oct 30 02:41:16.209395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:16.209324    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tdffn"]
Oct 30 02:41:16.213126 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:16.213098    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6626491_0f7e_456e_b1ff_e8f93e2325cd.slice/crio-b341d5c82e9de198c324dfa787ca6fc25d92a136985ed45cdf59fe9477ed0410 WatchSource:0}: Error finding container b341d5c82e9de198c324dfa787ca6fc25d92a136985ed45cdf59fe9477ed0410: Status 404 returned error can't find the container with id b341d5c82e9de198c324dfa787ca6fc25d92a136985ed45cdf59fe9477ed0410
Oct 30 02:41:17.152470 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:17.152429    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn" event={"ID":"d6626491-0f7e-456e-b1ff-e8f93e2325cd","Type":"ContainerStarted","Data":"b341d5c82e9de198c324dfa787ca6fc25d92a136985ed45cdf59fe9477ed0410"}
Oct 30 02:41:18.156662 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:18.156624    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn" event={"ID":"d6626491-0f7e-456e-b1ff-e8f93e2325cd","Type":"ContainerStarted","Data":"667a12121591755e890e21c2ecadd8c11a3db3ccdfd26f59550cf27b78db5f9c"}
Oct 30 02:41:18.157007 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:18.156686    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn" event={"ID":"d6626491-0f7e-456e-b1ff-e8f93e2325cd","Type":"ContainerStarted","Data":"1f6a2017c9dd53cdd71c7a18d253dffe3fb322a6e314e2077c9f07a4ab0bce80"}
Oct 30 02:41:18.196803 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:18.196737    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tdffn" podStartSLOduration=2.225220743 podStartE2EDuration="3.196723849s" totalImagesPullingTime="971.503106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:18.196408545 +0000 UTC m=+17.270579339" watchObservedRunningTime="2026-10-30 02:41:18.196723849 +0000 UTC m=+17.270894673"
Oct 30 02:41:20.172157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.172130    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v56d"
Oct 30 02:41:20.330956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.330920    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5dnxm"]
Oct 30 02:41:20.333851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.333833    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.361398 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.361376    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:41:20.373673 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.373651    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:41:20.373755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.373652    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vvql5\""
Oct 30 02:41:20.440432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.440377    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:41:20.502426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502400    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-root\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502553 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502437    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-tls\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502553 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502457    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-sys\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502553 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502475    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-wtmp\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502571    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-textfile\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502601    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502629    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-metrics-client-ca\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502739    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.502811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.502784    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhs6\" (UniqueName: \"kubernetes.io/projected/20219a22-8edd-4c72-97dc-a2a35a1210e8-kube-api-access-lkhs6\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603794 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603766    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhs6\" (UniqueName: \"kubernetes.io/projected/20219a22-8edd-4c72-97dc-a2a35a1210e8-kube-api-access-lkhs6\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603799    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-root\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603832    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-tls\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603860    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-sys\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603884    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-wtmp\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603887    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-root\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.603914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603911    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-textfile\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603942    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603962    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-sys\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.603974    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-metrics-client-ca\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.604029    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.604065    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-wtmp\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604388 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.604302    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-textfile\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.604589    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.604684 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.604650    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20219a22-8edd-4c72-97dc-a2a35a1210e8-metrics-client-ca\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.606179 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.606158    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.606282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.606265    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20219a22-8edd-4c72-97dc-a2a35a1210e8-node-exporter-tls\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.625870 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.625846    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhs6\" (UniqueName: \"kubernetes.io/projected/20219a22-8edd-4c72-97dc-a2a35a1210e8-kube-api-access-lkhs6\") pod \"node-exporter-5dnxm\" (UID: \"20219a22-8edd-4c72-97dc-a2a35a1210e8\") " pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.646526 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:20.646507    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5dnxm"
Oct 30 02:41:20.655198 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:20.655173    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20219a22_8edd_4c72_97dc_a2a35a1210e8.slice/crio-2b91f054bd4a48ca6ea080edeb1f38406878414e338131847cf8ce83bc782195 WatchSource:0}: Error finding container 2b91f054bd4a48ca6ea080edeb1f38406878414e338131847cf8ce83bc782195: Status 404 returned error can't find the container with id 2b91f054bd4a48ca6ea080edeb1f38406878414e338131847cf8ce83bc782195
Oct 30 02:41:21.167173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.167134    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dnxm" event={"ID":"20219a22-8edd-4c72-97dc-a2a35a1210e8","Type":"ContainerStarted","Data":"2b91f054bd4a48ca6ea080edeb1f38406878414e338131847cf8ce83bc782195"}
Oct 30 02:41:21.822701 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.822670    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:41:21.825288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.825263    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.859755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.859731    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 02:41:21.872024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.871944    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 02:41:21.872145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872047    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 02:41:21.872145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872124    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 02:41:21.872145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872132    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 02:41:21.872303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872146    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 02:41:21.872303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872158    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lqm4h\""
Oct 30 02:41:21.872303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872178    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 02:41:21.872303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872181    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 02:41:21.872303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.872207    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 02:41:21.873049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.873033    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:41:21.913885 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.913863    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.913989 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.913895    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.913989 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.913915    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.913989 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.913933    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.913988    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914127    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxvh\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914148    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914164    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914180    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914210    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914255    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914326    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:21.914381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:21.914349    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015348    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015381    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015560 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015409    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015589    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015631    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015660    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015698    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxvh\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015729    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015756    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015783    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015816    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.015861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015851    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.016018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015886    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.016018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.015989    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.016451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.016420    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.016723 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.016655    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.018612 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.018584    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.018721 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.018668    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.018930 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.018903    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.019020 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.018936    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.019296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.019281    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.019417 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.019386    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.019524 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.019460    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.019655 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.019630    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.020067 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.020051    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.038251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.038219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxvh\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\") pod \"alertmanager-main-0\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.141792 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.141731    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:41:22.172798 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.172769    4214 generic.go:356] "Generic (PLEG): container finished" podID="20219a22-8edd-4c72-97dc-a2a35a1210e8" containerID="8dbe9175986d7bc83451c81cbefecaac2137e0574960846bcbd2cef69ae0fe4f" exitCode=0
Oct 30 02:41:22.172948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.172823    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dnxm" event={"ID":"20219a22-8edd-4c72-97dc-a2a35a1210e8","Type":"ContainerDied","Data":"8dbe9175986d7bc83451c81cbefecaac2137e0574960846bcbd2cef69ae0fe4f"}
Oct 30 02:41:22.497956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:22.497886    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:41:22.500994 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:22.500936    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ee14fa_725b_44dd_84bb_d259e9546a45.slice/crio-c7c653d33dd9552486ef3b09ea206df3725c22db7a4a0374c7fcf1f01d75659e WatchSource:0}: Error finding container c7c653d33dd9552486ef3b09ea206df3725c22db7a4a0374c7fcf1f01d75659e: Status 404 returned error can't find the container with id c7c653d33dd9552486ef3b09ea206df3725c22db7a4a0374c7fcf1f01d75659e
Oct 30 02:41:23.177380 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:23.177344    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"c7c653d33dd9552486ef3b09ea206df3725c22db7a4a0374c7fcf1f01d75659e"}
Oct 30 02:41:23.179846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:23.179764    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dnxm" event={"ID":"20219a22-8edd-4c72-97dc-a2a35a1210e8","Type":"ContainerStarted","Data":"fb64441cb21573051232f0ab39e8b3537a88eec2f7065d33bfed16f02aa630a8"}
Oct 30 02:41:23.179846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:23.179815    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5dnxm" event={"ID":"20219a22-8edd-4c72-97dc-a2a35a1210e8","Type":"ContainerStarted","Data":"09e0ac03eeeb341abd02f32b31d24bafa35d58b08935d066e2049791e0bc7058"}
Oct 30 02:41:23.223888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:23.223838    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5dnxm" podStartSLOduration=2.304363612 podStartE2EDuration="3.223821292s" totalImagesPullingTime="852.475838ms" totalInitContainerRuntime="66.981842ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:23.222889025 +0000 UTC m=+22.297059814" watchObservedRunningTime="2026-10-30 02:41:23.223821292 +0000 UTC m=+22.297992087"
Oct 30 02:41:24.184176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:24.184143    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402" exitCode=0
Oct 30 02:41:24.184657 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:24.184255    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"}
Oct 30 02:41:25.025368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.025332    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-779ms"]
Oct 30 02:41:25.027749 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.027722    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.044436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.044410    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pst8k\""
Oct 30 02:41:25.063754 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.063725    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-779ms"]
Oct 30 02:41:25.063922 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.063773    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f76f89f7f-vw69h"]
Oct 30 02:41:25.065914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.065894    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.085404 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.085381    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 02:41:25.085542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.085447    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 02:41:25.091057 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.091027    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f76f89f7f-vw69h"]
Oct 30 02:41:25.139329 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.139293    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfaf4064-d29b-44e0-a681-2ea5e15cf178-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.139473 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.139353    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vs4\" (UniqueName: \"kubernetes.io/projected/cfaf4064-d29b-44e0-a681-2ea5e15cf178-kube-api-access-42vs4\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.146747 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.146717    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 02:41:25.147131 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.147110    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 02:41:25.147281 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.147266    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c51goe8f9v7gm\""
Oct 30 02:41:25.147334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.147295    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 02:41:25.147334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.147323    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b7wsl\""
Oct 30 02:41:25.240201 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240162    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-client-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240217    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240352    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfaf4064-d29b-44e0-a681-2ea5e15cf178-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240435    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsm5c\" (UniqueName: \"kubernetes.io/projected/8e2a7555-f02f-4c87-8a50-6e8699c1815b-kube-api-access-rsm5c\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240467    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e2a7555-f02f-4c87-8a50-6e8699c1815b-audit-log\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240500    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vs4\" (UniqueName: \"kubernetes.io/projected/cfaf4064-d29b-44e0-a681-2ea5e15cf178-kube-api-access-42vs4\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.240686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240553    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-metrics-server-audit-profiles\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240690    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-tls\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.240924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.240747    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-client-certs\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.243020 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.242998    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfaf4064-d29b-44e0-a681-2ea5e15cf178-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.261153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.261128    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vs4\" (UniqueName: \"kubernetes.io/projected/cfaf4064-d29b-44e0-a681-2ea5e15cf178-kube-api-access-42vs4\") pod \"monitoring-plugin-77f4f76594-779ms\" (UID: \"cfaf4064-d29b-44e0-a681-2ea5e15cf178\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.339871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.339847    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:25.341082 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341058    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsm5c\" (UniqueName: \"kubernetes.io/projected/8e2a7555-f02f-4c87-8a50-6e8699c1815b-kube-api-access-rsm5c\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341096    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e2a7555-f02f-4c87-8a50-6e8699c1815b-audit-log\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341115    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-metrics-server-audit-profiles\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341144    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-tls\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341345 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341182    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-client-certs\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341345 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341222    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-client-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341345 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341269    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341516    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e2a7555-f02f-4c87-8a50-6e8699c1815b-audit-log\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.341917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341894    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.342022 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.341983    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e2a7555-f02f-4c87-8a50-6e8699c1815b-metrics-server-audit-profiles\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.343717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.343686    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-tls\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.343794 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.343731    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-client-ca-bundle\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.343829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.343812    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e2a7555-f02f-4c87-8a50-6e8699c1815b-secret-metrics-server-client-certs\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.361591 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.361560    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsm5c\" (UniqueName: \"kubernetes.io/projected/8e2a7555-f02f-4c87-8a50-6e8699c1815b-kube-api-access-rsm5c\") pod \"metrics-server-5f76f89f7f-vw69h\" (UID: \"8e2a7555-f02f-4c87-8a50-6e8699c1815b\") " pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.377025 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.376989    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:25.641390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.641366    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-779ms"]
Oct 30 02:41:25.644857 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:25.644820    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaf4064_d29b_44e0_a681_2ea5e15cf178.slice/crio-5ada5acf53fa0bf54b6ad38478ba29a122b41323a5087061dd624633622df584 WatchSource:0}: Error finding container 5ada5acf53fa0bf54b6ad38478ba29a122b41323a5087061dd624633622df584: Status 404 returned error can't find the container with id 5ada5acf53fa0bf54b6ad38478ba29a122b41323a5087061dd624633622df584
Oct 30 02:41:25.697174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:25.697135    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f76f89f7f-vw69h"]
Oct 30 02:41:25.700607 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:25.700583    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2a7555_f02f_4c87_8a50_6e8699c1815b.slice/crio-71b9810251e92bbb6ecb0acc8242f9cfbe39fdb445be7370ff37488bd4f1650a WatchSource:0}: Error finding container 71b9810251e92bbb6ecb0acc8242f9cfbe39fdb445be7370ff37488bd4f1650a: Status 404 returned error can't find the container with id 71b9810251e92bbb6ecb0acc8242f9cfbe39fdb445be7370ff37488bd4f1650a
Oct 30 02:41:26.197948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.197915    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"}
Oct 30 02:41:26.198140 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.198001    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"}
Oct 30 02:41:26.198140 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.198049    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"}
Oct 30 02:41:26.198140 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.198088    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"}
Oct 30 02:41:26.198140 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.198130    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"}
Oct 30 02:41:26.200355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.200325    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h" event={"ID":"8e2a7555-f02f-4c87-8a50-6e8699c1815b","Type":"ContainerStarted","Data":"71b9810251e92bbb6ecb0acc8242f9cfbe39fdb445be7370ff37488bd4f1650a"}
Oct 30 02:41:26.202091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:26.201990    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms" event={"ID":"cfaf4064-d29b-44e0-a681-2ea5e15cf178","Type":"ContainerStarted","Data":"5ada5acf53fa0bf54b6ad38478ba29a122b41323a5087061dd624633622df584"}
Oct 30 02:41:27.211520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.211431    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms" event={"ID":"cfaf4064-d29b-44e0-a681-2ea5e15cf178","Type":"ContainerStarted","Data":"c60399733f51de977f6521c98a3c9dc314ee5323ec02e3dbda75d0b53c1d963c"}
Oct 30 02:41:27.212013 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.211963    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:27.216373 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.216345    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerStarted","Data":"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"}
Oct 30 02:41:27.217872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.217845    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms"
Oct 30 02:41:27.256481 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.256435    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-779ms" podStartSLOduration=1.953570983 podStartE2EDuration="3.256420395s" totalImagesPullingTime="1.302849412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:27.25585443 +0000 UTC m=+26.330025224" watchObservedRunningTime="2026-10-30 02:41:27.256420395 +0000 UTC m=+26.330591185"
Oct 30 02:41:27.412214 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:27.412157    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.220428432 podStartE2EDuration="6.412137319s" totalImagesPullingTime="3.181014149s" totalInitContainerRuntime="10.694738ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:27.411735436 +0000 UTC m=+26.485906228" watchObservedRunningTime="2026-10-30 02:41:27.412137319 +0000 UTC m=+26.486308113"
Oct 30 02:41:28.220724 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:28.220680    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h" event={"ID":"8e2a7555-f02f-4c87-8a50-6e8699c1815b","Type":"ContainerStarted","Data":"be4e48732d3d612f25caa1e89e0e69443da993c4266452c65166cd46ba238585"}
Oct 30 02:41:28.265935 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:28.265880    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h" podStartSLOduration=1.698383612 podStartE2EDuration="3.26586592s" totalImagesPullingTime="1.567482308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:28.265475249 +0000 UTC m=+27.339646042" watchObservedRunningTime="2026-10-30 02:41:28.26586592 +0000 UTC m=+27.340036713"
Oct 30 02:41:28.984107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:28.984075    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:41:28.986540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:28.986524    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.022813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.022780    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:41:29.022981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.022831    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:41:29.022981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.022850    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:41:29.022981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.022899    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:41:29.022981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.022901    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:41:29.036527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.036505    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:41:29.036652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.036544    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:41:29.049899 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.049876    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fhbtn\""
Oct 30 02:41:29.050212 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.050192    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:41:29.051030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.051016    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:41:29.174105 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174069    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174105 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174106    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174137    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfsx\" (UniqueName: \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174305    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174330    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174476 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174355    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174476 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174373    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.174540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.174496    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.275839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275760    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.275839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275791    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.275839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275814    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.275839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275831    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275852    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.275992    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.276028    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.276064    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfsx\" (UniqueName: \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.276616    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.276649    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.276778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.276759    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.277053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.277037    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.277083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.277038    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.278192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.278168    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.278395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.278375    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.296375 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.296353    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfsx\" (UniqueName: \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\") pod \"console-6f88dc6598-xc5c8\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") " pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.479589 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.479559    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8xf7s"
Oct 30 02:41:29.596759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.596730    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:29.895599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:29.895535    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:41:29.898463 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:41:29.898436    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cf84fb_bde3_4dfc_9aba_04a8c0eca373.slice/crio-786106056e11170dcb1873a2f3d7fc4f0892cf6054ff4f52e471f47aa41fa152 WatchSource:0}: Error finding container 786106056e11170dcb1873a2f3d7fc4f0892cf6054ff4f52e471f47aa41fa152: Status 404 returned error can't find the container with id 786106056e11170dcb1873a2f3d7fc4f0892cf6054ff4f52e471f47aa41fa152
Oct 30 02:41:30.232031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:30.231960    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88dc6598-xc5c8" event={"ID":"56cf84fb-bde3-4dfc-9aba-04a8c0eca373","Type":"ContainerStarted","Data":"786106056e11170dcb1873a2f3d7fc4f0892cf6054ff4f52e471f47aa41fa152"}
Oct 30 02:41:33.245722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:33.245686    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88dc6598-xc5c8" event={"ID":"56cf84fb-bde3-4dfc-9aba-04a8c0eca373","Type":"ContainerStarted","Data":"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"}
Oct 30 02:41:33.287456 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:33.287394    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f88dc6598-xc5c8" podStartSLOduration=2.417442989 podStartE2EDuration="5.287378789s" totalImagesPullingTime="2.8699358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:33.287113209 +0000 UTC m=+32.361284001" watchObservedRunningTime="2026-10-30 02:41:33.287378789 +0000 UTC m=+32.361549581"
Oct 30 02:41:33.552469 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:33.552437    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:41:35.148675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:35.148649    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5677cfbdd5-87kq2"
Oct 30 02:41:39.162153 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:41:39.162115    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:41:39.597452 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:39.597415    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:41:45.377961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:45.377848    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:41:45.377961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:41:45.377909    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:42:00.271963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.271895    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f88dc6598-xc5c8" podUID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" containerName="console" containerID="cri-o://a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182" gracePeriod=15
Oct 30 02:42:00.505272 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.505249    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:42:00.603858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603821    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.603858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603864    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603887    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603910    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603936    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\" (UniqueName: \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603968    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.603986    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604012    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\") pod \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\" (UID: \"56cf84fb-bde3-4dfc-9aba-04a8c0eca373\") "
Oct 30 02:42:00.604507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604381    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.604507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604474    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.604507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604491    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.604647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604622    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:00.604821 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.604803    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.606146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.606123    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:00.606253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.606169    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:00.606253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.606173    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx" pod "56cf84fb-bde3-4dfc-9aba-04a8c0eca373" (UID: "56cf84fb-bde3-4dfc-9aba-04a8c0eca373"). InnerVolumeSpecName "kube-api-access-ncfsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:00.705508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705468    4214 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-serving-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705501    4214 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-config\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705510    4214 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-trusted-ca-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705519    4214 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-tmp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705530    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncfsx\" (UniqueName: \"kubernetes.io/projected/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-kube-api-access-ncfsx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705539    4214 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705547    4214 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-console-oauth-config\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.705743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:00.705555    4214 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56cf84fb-bde3-4dfc-9aba-04a8c0eca373-oauth-serving-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:01.124335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.124314    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7k44l_3e1acad5-796c-4ea8-b791-ce810c9438dd/serve-healthcheck-canary/0.log"
Oct 30 02:42:01.335851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.335773    4214 generic.go:356] "Generic (PLEG): container finished" podID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" containerID="a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182" exitCode=0
Oct 30 02:42:01.335851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.335825    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f88dc6598-xc5c8"
Oct 30 02:42:01.335851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.335831    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88dc6598-xc5c8" event={"ID":"56cf84fb-bde3-4dfc-9aba-04a8c0eca373","Type":"ContainerDied","Data":"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"}
Oct 30 02:42:01.336366 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.335895    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f88dc6598-xc5c8" event={"ID":"56cf84fb-bde3-4dfc-9aba-04a8c0eca373","Type":"ContainerDied","Data":"786106056e11170dcb1873a2f3d7fc4f0892cf6054ff4f52e471f47aa41fa152"}
Oct 30 02:42:01.336366 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.335922    4214 scope.go:122] "RemoveContainer" containerID="a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"
Oct 30 02:42:01.343524 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.343504    4214 scope.go:122] "RemoveContainer" containerID="a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"
Oct 30 02:42:01.343795 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:01.343774    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182\": container with ID starting with a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182 not found: ID does not exist" containerID="a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"
Oct 30 02:42:01.343881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.343801    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182"} err="failed to get container status \"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182\": rpc error: code = NotFound desc = could not find container \"a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182\": container with ID starting with a65a5e3336c3fb4bd0a4e8588b4e69c6b6dd1bdf7e4dfe56ffa68dbff26bf182 not found: ID does not exist"
Oct 30 02:42:01.389447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.389419    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:42:01.402213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:01.402188    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f88dc6598-xc5c8"]
Oct 30 02:42:03.063025 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:03.062992    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" path="/var/lib/kubelet/pods/56cf84fb-bde3-4dfc-9aba-04a8c0eca373/volumes"
Oct 30 02:42:05.382683 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:05.382655    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:42:05.386313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:05.386294    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f76f89f7f-vw69h"
Oct 30 02:42:09.162691 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:09.162656    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:42:39.161582 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:39.161548    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:42:41.974977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.974943    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:41.975365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975307    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="alertmanager" containerID="cri-o://20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1" gracePeriod=120
Oct 30 02:42:41.975437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975335    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-metric" containerID="cri-o://2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32" gracePeriod=120
Oct 30 02:42:41.975437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975387    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-web" containerID="cri-o://e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f" gracePeriod=120
Oct 30 02:42:41.975539 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975415    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy" containerID="cri-o://a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd" gracePeriod=120
Oct 30 02:42:41.975539 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975439    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="config-reloader" containerID="cri-o://f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e" gracePeriod=120
Oct 30 02:42:41.975539 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:41.975391    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="prom-label-proxy" containerID="cri-o://ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416" gracePeriod=120
Oct 30 02:42:42.467682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467649    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416" exitCode=0
Oct 30 02:42:42.467682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467673    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd" exitCode=0
Oct 30 02:42:42.467682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467679    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e" exitCode=0
Oct 30 02:42:42.467682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467684    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1" exitCode=0
Oct 30 02:42:42.467949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467718    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"}
Oct 30 02:42:42.467949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467780    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"}
Oct 30 02:42:42.467949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467808    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"}
Oct 30 02:42:42.467949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:42.467834    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"}
Oct 30 02:42:43.218692 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.218670    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.336645 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336616    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.336820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336657    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.336820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336692    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.336820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336722    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.336820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336765    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.336820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336798    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336823    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336859    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336884    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336930    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.336964    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.337017    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.337054    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\") pod \"25ee14fa-725b-44dd-84bb-d259e9546a45\" (UID: \"25ee14fa-725b-44dd-84bb-d259e9546a45\") "
Oct 30 02:42:43.337086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.337056    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:43.337543 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.337281    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-metrics-client-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.337929 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.337896    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:43.338180 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.338153    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:43.341171 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341141    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:43.341318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341176    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.341405 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341378    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:43.341509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341445    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.341686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341662    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.341829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.341812    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.342123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.342104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "kube-api-access-6gxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:43.342195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.342106    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.344193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.344171    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.349722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.349704    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config" pod "25ee14fa-725b-44dd-84bb-d259e9546a45" (UID: "25ee14fa-725b-44dd-84bb-d259e9546a45"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:43.438342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438318    4214 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-tls-assets\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438342    4214 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-config-out\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438355    4214 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-cluster-tls-config\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438369    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxvh\" (UniqueName: \"kubernetes.io/projected/25ee14fa-725b-44dd-84bb-d259e9546a45-kube-api-access-6gxvh\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438384    4214 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438397    4214 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438411    4214 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438424    4214 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438436    4214 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-web-config\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438448    4214 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25ee14fa-725b-44dd-84bb-d259e9546a45-alertmanager-main-db\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438460    4214 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-secret-alertmanager-main-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.438533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.438471    4214 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25ee14fa-725b-44dd-84bb-d259e9546a45-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:42:43.474974 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.474946    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32" exitCode=0
Oct 30 02:42:43.474974 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.474975    4214 generic.go:356] "Generic (PLEG): container finished" podID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerID="e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f" exitCode=0
Oct 30 02:42:43.475126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.475027    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"}
Oct 30 02:42:43.475126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.475042    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.475126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.475107    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"}
Oct 30 02:42:43.475220 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.475137    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25ee14fa-725b-44dd-84bb-d259e9546a45","Type":"ContainerDied","Data":"c7c653d33dd9552486ef3b09ea206df3725c22db7a4a0374c7fcf1f01d75659e"}
Oct 30 02:42:43.475220 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.475167    4214 scope.go:122] "RemoveContainer" containerID="ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"
Oct 30 02:42:43.484013 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.483998    4214 scope.go:122] "RemoveContainer" containerID="2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"
Oct 30 02:42:43.490508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.490482    4214 scope.go:122] "RemoveContainer" containerID="a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"
Oct 30 02:42:43.497174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.497154    4214 scope.go:122] "RemoveContainer" containerID="e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"
Oct 30 02:42:43.503313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.503299    4214 scope.go:122] "RemoveContainer" containerID="f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"
Oct 30 02:42:43.509494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.509474    4214 scope.go:122] "RemoveContainer" containerID="20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"
Oct 30 02:42:43.515525 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.515509    4214 scope.go:122] "RemoveContainer" containerID="3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"
Oct 30 02:42:43.521489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.521467    4214 scope.go:122] "RemoveContainer" containerID="ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"
Oct 30 02:42:43.521748 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.521727    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416\": container with ID starting with ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416 not found: ID does not exist" containerID="ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"
Oct 30 02:42:43.521797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.521757    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"} err="failed to get container status \"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416\": rpc error: code = NotFound desc = could not find container \"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416\": container with ID starting with ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416 not found: ID does not exist"
Oct 30 02:42:43.521797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.521774    4214 scope.go:122] "RemoveContainer" containerID="2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"
Oct 30 02:42:43.521981 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.521965    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32\": container with ID starting with 2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32 not found: ID does not exist" containerID="2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"
Oct 30 02:42:43.522018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.521987    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"} err="failed to get container status \"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32\": rpc error: code = NotFound desc = could not find container \"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32\": container with ID starting with 2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32 not found: ID does not exist"
Oct 30 02:42:43.522018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522003    4214 scope.go:122] "RemoveContainer" containerID="a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"
Oct 30 02:42:43.522191 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.522178    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd\": container with ID starting with a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd not found: ID does not exist" containerID="a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"
Oct 30 02:42:43.522323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522195    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"} err="failed to get container status \"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd\": rpc error: code = NotFound desc = could not find container \"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd\": container with ID starting with a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd not found: ID does not exist"
Oct 30 02:42:43.522323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522206    4214 scope.go:122] "RemoveContainer" containerID="e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"
Oct 30 02:42:43.522476 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.522461    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f\": container with ID starting with e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f not found: ID does not exist" containerID="e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"
Oct 30 02:42:43.522519 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522479    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"} err="failed to get container status \"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f\": rpc error: code = NotFound desc = could not find container \"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f\": container with ID starting with e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f not found: ID does not exist"
Oct 30 02:42:43.522519 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522491    4214 scope.go:122] "RemoveContainer" containerID="f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"
Oct 30 02:42:43.522664 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.522646    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e\": container with ID starting with f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e not found: ID does not exist" containerID="f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"
Oct 30 02:42:43.522759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522663    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"} err="failed to get container status \"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e\": rpc error: code = NotFound desc = could not find container \"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e\": container with ID starting with f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e not found: ID does not exist"
Oct 30 02:42:43.522759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522674    4214 scope.go:122] "RemoveContainer" containerID="20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"
Oct 30 02:42:43.522880 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.522866    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1\": container with ID starting with 20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1 not found: ID does not exist" containerID="20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"
Oct 30 02:42:43.522917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522882    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"} err="failed to get container status \"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1\": rpc error: code = NotFound desc = could not find container \"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1\": container with ID starting with 20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1 not found: ID does not exist"
Oct 30 02:42:43.522917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.522893    4214 scope.go:122] "RemoveContainer" containerID="3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"
Oct 30 02:42:43.523094 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:42:43.523071    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402\": container with ID starting with 3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402 not found: ID does not exist" containerID="3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"
Oct 30 02:42:43.523133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523097    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"} err="failed to get container status \"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402\": rpc error: code = NotFound desc = could not find container \"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402\": container with ID starting with 3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402 not found: ID does not exist"
Oct 30 02:42:43.523133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523107    4214 scope.go:122] "RemoveContainer" containerID="ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"
Oct 30 02:42:43.523316 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523300    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416"} err="failed to get container status \"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416\": rpc error: code = NotFound desc = could not find container \"ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416\": container with ID starting with ae92e6d2d7d5185cf9ac955a7210e48f7c6bb7a6c6df1b1ad458c8b36ef16416 not found: ID does not exist"
Oct 30 02:42:43.523363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523315    4214 scope.go:122] "RemoveContainer" containerID="2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"
Oct 30 02:42:43.523509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523494    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32"} err="failed to get container status \"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32\": rpc error: code = NotFound desc = could not find container \"2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32\": container with ID starting with 2283a4bbcdb22d02667ebce3f797a666f3b2fbf1b4e0d858f280b63fcbc7bf32 not found: ID does not exist"
Oct 30 02:42:43.523554 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523510    4214 scope.go:122] "RemoveContainer" containerID="a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"
Oct 30 02:42:43.523730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523711    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd"} err="failed to get container status \"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd\": rpc error: code = NotFound desc = could not find container \"a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd\": container with ID starting with a3bcd008639612b163e276f0048fe82cddab9a1b08bdfb178852fedd3d0174dd not found: ID does not exist"
Oct 30 02:42:43.523730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523729    4214 scope.go:122] "RemoveContainer" containerID="e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"
Oct 30 02:42:43.523906 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523891    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f"} err="failed to get container status \"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f\": rpc error: code = NotFound desc = could not find container \"e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f\": container with ID starting with e8186d3381feebb8146b94592d6d2622d8fc939ef10150264d7be7fcd905739f not found: ID does not exist"
Oct 30 02:42:43.523906 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.523906    4214 scope.go:122] "RemoveContainer" containerID="f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"
Oct 30 02:42:43.524078 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.524063    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e"} err="failed to get container status \"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e\": rpc error: code = NotFound desc = could not find container \"f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e\": container with ID starting with f0345c5f103ddc4f36b6ef68fae34c13cbedf3d8118af5879d6018995c52b59e not found: ID does not exist"
Oct 30 02:42:43.524118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.524077    4214 scope.go:122] "RemoveContainer" containerID="20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"
Oct 30 02:42:43.524273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.524257    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1"} err="failed to get container status \"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1\": rpc error: code = NotFound desc = could not find container \"20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1\": container with ID starting with 20667e49b091450dda179039dde9cdd3e3626d15b255b39c5eaa094e13e2ffa1 not found: ID does not exist"
Oct 30 02:42:43.524322 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.524273    4214 scope.go:122] "RemoveContainer" containerID="3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"
Oct 30 02:42:43.524454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.524435    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402"} err="failed to get container status \"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402\": rpc error: code = NotFound desc = could not find container \"3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402\": container with ID starting with 3ced434c2d4b24c6276036f9c5c5df9ec289ee30a0d5ea28d26c119f02bf5402 not found: ID does not exist"
Oct 30 02:42:43.530040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.530022    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:43.533688 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.533671    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:43.559890 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.559867    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:43.560084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560072    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-web"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560085    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-web"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560099    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560103    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560110    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="prom-label-proxy"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560115    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="prom-label-proxy"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560121    4214 cpu_manager.go:414] "containerMap: removing container" podUID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" containerName="console"
Oct 30 02:42:43.560124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560126    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" containerName="console"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560138    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="init-config-reloader"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560142    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="init-config-reloader"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560148    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="alertmanager"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560152    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="alertmanager"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560157    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="config-reloader"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560161    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="config-reloader"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560171    4214 cpu_manager.go:414] "containerMap: removing container" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-metric"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560176    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-metric"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560211    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="config-reloader"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560218    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="alertmanager"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560224    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-metric"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560244    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy-web"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560250    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="kube-rbac-proxy"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560255    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" containerName="prom-label-proxy"
Oct 30 02:42:43.560337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.560259    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cf84fb-bde3-4dfc-9aba-04a8c0eca373" containerName="console"
Oct 30 02:42:43.566398 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.566381    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.582086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.582068    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 02:42:43.582188 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.582173    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 02:42:43.582296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.582276    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 02:42:43.582850 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.582817    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 02:42:43.586693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.583103    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 02:42:43.586693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.583938    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 02:42:43.586693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.585409    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 02:42:43.586693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.585446    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lqm4h\""
Oct 30 02:42:43.586693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.585480    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 02:42:43.595191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.595170    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 02:42:43.596730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.596710    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:43.740355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740323    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740359    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740380    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740441    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740466    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740493    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740555    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740607    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740628    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740717    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740876 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740774    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740876 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740794    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.740876 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.740814    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzmx\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-kube-api-access-zzzmx\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841373    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841409    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841426    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841462    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841487    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841512    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841539    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841556    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841597    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841623    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.841664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841656    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzmx\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-kube-api-access-zzzmx\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.842041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841701    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.842041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.841728    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.842265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.842217    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.842440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.842419    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844562 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844535    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844660 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844631    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844660 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844639    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844772 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844660    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844772 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844694    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.844843 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.844797    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.845103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.845081    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.845169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.845150    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.845293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.845277    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.846576 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.846556    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.861952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.861932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzmx\" (UniqueName: \"kubernetes.io/projected/a9cab9a6-c144-4af0-94fb-ac7ecb1f396e-kube-api-access-zzzmx\") pod \"alertmanager-main-0\" (UID: \"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:43.890815 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:43.890794    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:42:44.248179 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:44.247887    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:42:44.250428 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:42:44.250399    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cab9a6_c144_4af0_94fb_ac7ecb1f396e.slice/crio-e0b9de67825ffb1f83fa99f0d003b6235b9f028f1bf0966c1710cd0c3b2582bf WatchSource:0}: Error finding container e0b9de67825ffb1f83fa99f0d003b6235b9f028f1bf0966c1710cd0c3b2582bf: Status 404 returned error can't find the container with id e0b9de67825ffb1f83fa99f0d003b6235b9f028f1bf0966c1710cd0c3b2582bf
Oct 30 02:42:44.480759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:44.480721    4214 generic.go:356] "Generic (PLEG): container finished" podID="a9cab9a6-c144-4af0-94fb-ac7ecb1f396e" containerID="1c751cb7b9912bada0d45ef24f081a054f942017dc0f60ac9a68cf4e121d04f8" exitCode=0
Oct 30 02:42:44.480977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:44.480801    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerDied","Data":"1c751cb7b9912bada0d45ef24f081a054f942017dc0f60ac9a68cf4e121d04f8"}
Oct 30 02:42:44.480977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:44.480870    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"e0b9de67825ffb1f83fa99f0d003b6235b9f028f1bf0966c1710cd0c3b2582bf"}
Oct 30 02:42:45.063088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.063049    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ee14fa-725b-44dd-84bb-d259e9546a45" path="/var/lib/kubelet/pods/25ee14fa-725b-44dd-84bb-d259e9546a45/volumes"
Oct 30 02:42:45.487250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487199    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"8ed89aef6a26355cce869d2fc2f6d184ced26ac2ffe7209b7b9313f58803fc5f"}
Oct 30 02:42:45.487636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487297    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"03b5acb57f2f149ee53e7b1f20bb071132c80a36f6dae801a4b7895a6f4fef20"}
Oct 30 02:42:45.487636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487344    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"f611405eaab81b6b17bee67d1ac6785385dc81c3d5a52a513ba94c28c935bb73"}
Oct 30 02:42:45.487636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487378    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"9889031dcba815cd57f3b7543ae43b4670ff0284385f8eacd0f12b5844a031d2"}
Oct 30 02:42:45.487636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487415    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"d9102170e75f9dc7e6b97153cecc0a85b9172ebbee1f0bb2c38b26bfb130d356"}
Oct 30 02:42:45.487636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.487449    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cab9a6-c144-4af0-94fb-ac7ecb1f396e","Type":"ContainerStarted","Data":"075038b2055c780b0f87319958858714e0a63a13b56e6205baed90f2a6c713b0"}
Oct 30 02:42:45.573583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:42:45.573521    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5627970429999998 podStartE2EDuration="2.57350581s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.708767ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:45.560604207 +0000 UTC m=+104.634775000" watchObservedRunningTime="2026-10-30 02:42:45.57350581 +0000 UTC m=+104.647676599"
Oct 30 02:43:09.161623 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:43:09.161587    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:43:39.162115 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:43:39.162070    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:44:09.162438 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:44:09.162397    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:44:39.162310 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:44:39.162272    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:44:54.511671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.511597    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"]
Oct 30 02:44:54.514849 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.514833    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.545699 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.545672    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 02:44:54.545871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.545702    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 02:44:54.545871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.545714    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kt68t\""
Oct 30 02:44:54.546755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.546732    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"]
Oct 30 02:44:54.584949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.584912    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.584949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.584952    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.585194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.584971    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmv6\" (UniqueName: \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.686310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.686274    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.686509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.686318    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.686509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.686345    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmv6\" (UniqueName: \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.686688 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.686665    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.686747 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.686701    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.708542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.708509    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmv6\" (UniqueName: \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:54.829992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:54.829956    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:44:55.126789 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:55.126716    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"]
Oct 30 02:44:55.130142 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:44:55.130116    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fddc148_df10_4694_9a0b_4b035c2aeab3.slice/crio-315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2 WatchSource:0}: Error finding container 315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2: Status 404 returned error can't find the container with id 315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2
Oct 30 02:44:55.901186 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:44:55.901152    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq" event={"ID":"5fddc148-df10-4694-9a0b-4b035c2aeab3","Type":"ContainerStarted","Data":"315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2"}
Oct 30 02:45:00.921682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:00.921653    4214 generic.go:356] "Generic (PLEG): container finished" podID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerID="f251e444e3651c60cb47274f6091a1fe1d8361bb29cd3be71c079dbdf25d7652" exitCode=0
Oct 30 02:45:00.922163 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:00.921743    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq" event={"ID":"5fddc148-df10-4694-9a0b-4b035c2aeab3","Type":"ContainerDied","Data":"f251e444e3651c60cb47274f6091a1fe1d8361bb29cd3be71c079dbdf25d7652"}
Oct 30 02:45:03.934530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:03.934494    4214 generic.go:356] "Generic (PLEG): container finished" podID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerID="d7e8bea6e1d0b8ea7af0bf50c0552cbc5b01094250b1b47c89ef34f8d20ccfbd" exitCode=0
Oct 30 02:45:03.934898 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:03.934582    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq" event={"ID":"5fddc148-df10-4694-9a0b-4b035c2aeab3","Type":"ContainerDied","Data":"d7e8bea6e1d0b8ea7af0bf50c0552cbc5b01094250b1b47c89ef34f8d20ccfbd"}
Oct 30 02:45:09.165583 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:45:09.165544    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:45:09.959349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:09.959317    4214 generic.go:356] "Generic (PLEG): container finished" podID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerID="1882c976a6e44ab7630ca33475163940e294550cca02c8831123dee05b3756b5" exitCode=0
Oct 30 02:45:09.959550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:09.959406    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq" event={"ID":"5fddc148-df10-4694-9a0b-4b035c2aeab3","Type":"ContainerDied","Data":"1882c976a6e44ab7630ca33475163940e294550cca02c8831123dee05b3756b5"}
Oct 30 02:45:11.084918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.084891    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:45:11.224906 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.224817    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\") pod \"5fddc148-df10-4694-9a0b-4b035c2aeab3\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") "
Oct 30 02:45:11.224906 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.224871    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\") pod \"5fddc148-df10-4694-9a0b-4b035c2aeab3\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") "
Oct 30 02:45:11.225153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.224946    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\" (UniqueName: \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\") pod \"5fddc148-df10-4694-9a0b-4b035c2aeab3\" (UID: \"5fddc148-df10-4694-9a0b-4b035c2aeab3\") "
Oct 30 02:45:11.225584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.225545    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle" pod "5fddc148-df10-4694-9a0b-4b035c2aeab3" (UID: "5fddc148-df10-4694-9a0b-4b035c2aeab3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:11.227139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.227110    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6" pod "5fddc148-df10-4694-9a0b-4b035c2aeab3" (UID: "5fddc148-df10-4694-9a0b-4b035c2aeab3"). InnerVolumeSpecName "kube-api-access-mwmv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:45:11.228840 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.228810    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util" pod "5fddc148-df10-4694-9a0b-4b035c2aeab3" (UID: "5fddc148-df10-4694-9a0b-4b035c2aeab3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:11.326292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.326220    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmv6\" (UniqueName: \"kubernetes.io/projected/5fddc148-df10-4694-9a0b-4b035c2aeab3-kube-api-access-mwmv6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:11.326292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.326285    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:11.326292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.326297    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fddc148-df10-4694-9a0b-4b035c2aeab3-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:11.967559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.967522    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq" event={"ID":"5fddc148-df10-4694-9a0b-4b035c2aeab3","Type":"ContainerDied","Data":"315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2"}
Oct 30 02:45:11.967559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.967552    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpkkrq"
Oct 30 02:45:11.967766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:11.967577    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315dbaee9b3d7b7ab050a1e98f29d2ca350efacaa1b9d5a4b5e2d63a843297a2"
Oct 30 02:45:17.382701 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382664    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"]
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382932    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="util"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382944    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="util"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382960    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="pull"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382964    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="pull"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382978    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="extract"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.382983    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="extract"
Oct 30 02:45:17.383108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.383025    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fddc148-df10-4694-9a0b-4b035c2aeab3" containerName="extract"
Oct 30 02:45:17.432178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.432140    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"]
Oct 30 02:45:17.432355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.432160    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.448395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.448369    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:45:17.448395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.448378    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 02:45:17.448560 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.448448    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2q8zd\""
Oct 30 02:45:17.574477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.574430    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whcf\" (UniqueName: \"kubernetes.io/projected/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-kube-api-access-7whcf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.574665 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.574507    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.675786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.675699    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.675786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.675762    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whcf\" (UniqueName: \"kubernetes.io/projected/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-kube-api-access-7whcf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.676156 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.676130    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.695570 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.695541    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whcf\" (UniqueName: \"kubernetes.io/projected/e262f4cc-d57c-425b-a4fb-f8c3324fa9ba-kube-api-access-7whcf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-vcjd4\" (UID: \"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:17.754392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:17.754353    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"
Oct 30 02:45:18.103961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:18.103929    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4"]
Oct 30 02:45:18.121291 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:18.121262    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode262f4cc_d57c_425b_a4fb_f8c3324fa9ba.slice/crio-2f8d8027c5607cac12fb849bb5d42af2785284e7a81673c06a8b7e20182b50f3 WatchSource:0}: Error finding container 2f8d8027c5607cac12fb849bb5d42af2785284e7a81673c06a8b7e20182b50f3: Status 404 returned error can't find the container with id 2f8d8027c5607cac12fb849bb5d42af2785284e7a81673c06a8b7e20182b50f3
Oct 30 02:45:18.992262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:18.992215    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4" event={"ID":"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba","Type":"ContainerStarted","Data":"2f8d8027c5607cac12fb849bb5d42af2785284e7a81673c06a8b7e20182b50f3"}
Oct 30 02:45:21.006912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.006870    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4" event={"ID":"e262f4cc-d57c-425b-a4fb-f8c3324fa9ba","Type":"ContainerStarted","Data":"33229477a603f7d0b7cb4881f7d80d2b16a9cb1979af726e9d75a876f97ef6dd"}
Oct 30 02:45:21.076697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.076644    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-vcjd4" podStartSLOduration=1.726691465 podStartE2EDuration="4.076631148s" totalImagesPullingTime="2.349939683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:21.062806903 +0000 UTC m=+260.136977722" watchObservedRunningTime="2026-10-30 02:45:21.076631148 +0000 UTC m=+260.150801940"
Oct 30 02:45:21.778564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.778533    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"]
Oct 30 02:45:21.801693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.801665    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:21.802093 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.802066    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"]
Oct 30 02:45:21.818296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.818276    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 02:45:21.818393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.818316    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zxrjj\""
Oct 30 02:45:21.818393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.818316    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 02:45:21.840579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.840556    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"]
Oct 30 02:45:21.857978 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.857955    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:21.863365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.863339    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"]
Oct 30 02:45:21.913518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.913486    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvtl\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-kube-api-access-4xvtl\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:21.913696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.913535    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:21.918512 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.918477    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-zmbpm\""
Oct 30 02:45:21.918635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.918524    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 02:45:21.918635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:21.918525    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 02:45:22.014097 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.014068    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqpm\" (UniqueName: \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.014573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.014108    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:22.014573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.014139    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.014573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.014341    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.014573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.014397    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvtl\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-kube-api-access-4xvtl\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:22.038515 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.038449    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:22.038750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.038727    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvtl\" (UniqueName: \"kubernetes.io/projected/6ec5181a-fc48-4367-a7b6-b57fc2da69c1-kube-api-access-4xvtl\") pod \"cert-manager-webhook-5dbbbf75cc-xl95z\" (UID: \"6ec5181a-fc48-4367-a7b6-b57fc2da69c1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:22.115725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.115687    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.115903 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.115737    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqpm\" (UniqueName: \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.115903 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.115777    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.116182 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.116159    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.116219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.116190    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.132817 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.132790    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:22.138430 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.138396    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqpm\" (UniqueName: \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.212986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.212759    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:22.455930 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.455900    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"]
Oct 30 02:45:22.458327 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:22.458295    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ec5181a_fc48_4367_a7b6_b57fc2da69c1.slice/crio-ef80c2bdc1102df1099f2f91fb792674e2b57bb76d8fae7e6020a130d442add2 WatchSource:0}: Error finding container ef80c2bdc1102df1099f2f91fb792674e2b57bb76d8fae7e6020a130d442add2: Status 404 returned error can't find the container with id ef80c2bdc1102df1099f2f91fb792674e2b57bb76d8fae7e6020a130d442add2
Oct 30 02:45:22.520550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:22.520525    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"]
Oct 30 02:45:22.522739 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:22.522709    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75caf35_e627_4b1f_a928_e3e8dbefdf15.slice/crio-aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046 WatchSource:0}: Error finding container aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046: Status 404 returned error can't find the container with id aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046
Oct 30 02:45:23.015624 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:23.015586    4214 generic.go:356] "Generic (PLEG): container finished" podID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerID="c8e378c8908600de2174e36842323c0e4d02bc82c193d9eee72eea86bb6b8621" exitCode=0
Oct 30 02:45:23.016084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:23.015626    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26" event={"ID":"b75caf35-e627-4b1f-a928-e3e8dbefdf15","Type":"ContainerDied","Data":"c8e378c8908600de2174e36842323c0e4d02bc82c193d9eee72eea86bb6b8621"}
Oct 30 02:45:23.016084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:23.015680    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26" event={"ID":"b75caf35-e627-4b1f-a928-e3e8dbefdf15","Type":"ContainerStarted","Data":"aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046"}
Oct 30 02:45:23.016858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:23.016833    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z" event={"ID":"6ec5181a-fc48-4367-a7b6-b57fc2da69c1","Type":"ContainerStarted","Data":"ef80c2bdc1102df1099f2f91fb792674e2b57bb76d8fae7e6020a130d442add2"}
Oct 30 02:45:26.030825 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:26.030793    4214 generic.go:356] "Generic (PLEG): container finished" podID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerID="1e9be98ec49e2e7f61142b22a99f742a08a955c2c7bce3dbfa228479fb6b0117" exitCode=0
Oct 30 02:45:26.031310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:26.030881    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26" event={"ID":"b75caf35-e627-4b1f-a928-e3e8dbefdf15","Type":"ContainerDied","Data":"1e9be98ec49e2e7f61142b22a99f742a08a955c2c7bce3dbfa228479fb6b0117"}
Oct 30 02:45:26.032435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:26.032410    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z" event={"ID":"6ec5181a-fc48-4367-a7b6-b57fc2da69c1","Type":"ContainerStarted","Data":"eb3bc87e0c55a013baf6b82b1ed87d1c4ddab42c91f781926050d5342ba5b795"}
Oct 30 02:45:26.032559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:26.032546    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:26.109800 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:26.109728    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z" podStartSLOduration=2.099369153 podStartE2EDuration="5.109713127s" totalImagesPullingTime="3.010343974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:26.108819037 +0000 UTC m=+265.182989867" watchObservedRunningTime="2026-10-30 02:45:26.109713127 +0000 UTC m=+265.183883920"
Oct 30 02:45:27.038574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:27.038540    4214 generic.go:356] "Generic (PLEG): container finished" podID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerID="28730fe3f78f667f4206c4b7ac49bad5be65358852aae0630bbedb55dbe02e90" exitCode=0
Oct 30 02:45:27.039097 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:27.038623    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26" event={"ID":"b75caf35-e627-4b1f-a928-e3e8dbefdf15","Type":"ContainerDied","Data":"28730fe3f78f667f4206c4b7ac49bad5be65358852aae0630bbedb55dbe02e90"}
Oct 30 02:45:28.145124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.145098    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"]
Oct 30 02:45:28.147129 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.147112    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.160958 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.160938    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:28.163150 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.163124    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\") pod \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") "
Oct 30 02:45:28.163292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.163205    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\") pod \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") "
Oct 30 02:45:28.163364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.163336    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.163551 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.163375    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhmg\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-kube-api-access-tqhmg\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.163787 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.163763    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle" pod "b75caf35-e627-4b1f-a928-e3e8dbefdf15" (UID: "b75caf35-e627-4b1f-a928-e3e8dbefdf15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:28.165646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.165618    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"]
Oct 30 02:45:28.168743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.168719    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util" pod "b75caf35-e627-4b1f-a928-e3e8dbefdf15" (UID: "b75caf35-e627-4b1f-a928-e3e8dbefdf15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:28.177438 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.177421    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8z82n\""
Oct 30 02:45:28.263756 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.263729    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\" (UniqueName: \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\") pod \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\" (UID: \"b75caf35-e627-4b1f-a928-e3e8dbefdf15\") "
Oct 30 02:45:28.263912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.263828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.263912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.263850    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhmg\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-kube-api-access-tqhmg\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.263912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.263894    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:28.263912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.263908    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b75caf35-e627-4b1f-a928-e3e8dbefdf15-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:28.265788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.265760    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm" pod "b75caf35-e627-4b1f-a928-e3e8dbefdf15" (UID: "b75caf35-e627-4b1f-a928-e3e8dbefdf15"). InnerVolumeSpecName "kube-api-access-2mqpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:45:28.283492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.283468    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.283784 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.283763    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhmg\" (UniqueName: \"kubernetes.io/projected/9fd06c6b-2567-4c07-969f-c5416d9e3d35-kube-api-access-tqhmg\") pod \"cert-manager-cainjector-5c57bbd4fc-r5qms\" (UID: \"9fd06c6b-2567-4c07-969f-c5416d9e3d35\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.364588 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.364556    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mqpm\" (UniqueName: \"kubernetes.io/projected/b75caf35-e627-4b1f-a928-e3e8dbefdf15-kube-api-access-2mqpm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:28.468305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.468269    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"
Oct 30 02:45:28.759712 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:28.759639    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms"]
Oct 30 02:45:28.763142 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:28.763123    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fd06c6b_2567_4c07_969f_c5416d9e3d35.slice/crio-baf41b91b362d211b3c5080eec4bae75a5dfbcc1df14dc43be98cd5a90547638 WatchSource:0}: Error finding container baf41b91b362d211b3c5080eec4bae75a5dfbcc1df14dc43be98cd5a90547638: Status 404 returned error can't find the container with id baf41b91b362d211b3c5080eec4bae75a5dfbcc1df14dc43be98cd5a90547638
Oct 30 02:45:29.047877 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.047851    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26"
Oct 30 02:45:29.048065 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.047868    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lpc26" event={"ID":"b75caf35-e627-4b1f-a928-e3e8dbefdf15","Type":"ContainerDied","Data":"aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046"}
Oct 30 02:45:29.048065 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.047927    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aef2fb7acc75de050420de87c9862d4c7a086f53a49dfcdf81ffd8f140f67046"
Oct 30 02:45:29.049495 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.049471    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms" event={"ID":"9fd06c6b-2567-4c07-969f-c5416d9e3d35","Type":"ContainerStarted","Data":"255339c6572b74342f058f062a1e028b17e0aa85c561c881c71a17ef14441bef"}
Oct 30 02:45:29.049646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.049503    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms" event={"ID":"9fd06c6b-2567-4c07-969f-c5416d9e3d35","Type":"ContainerStarted","Data":"baf41b91b362d211b3c5080eec4bae75a5dfbcc1df14dc43be98cd5a90547638"}
Oct 30 02:45:29.094774 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:29.094723    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r5qms" podStartSLOduration=1.09470859 podStartE2EDuration="1.09470859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:29.093690136 +0000 UTC m=+268.167860931" watchObservedRunningTime="2026-10-30 02:45:29.09470859 +0000 UTC m=+268.168879382"
Oct 30 02:45:32.041136 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:32.041103    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z"
Oct 30 02:45:39.162553 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:45:39.162512    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:45:42.292693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.292658    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"]
Oct 30 02:45:42.293146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293089    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="extract"
Oct 30 02:45:42.293146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293106    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="extract"
Oct 30 02:45:42.293146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293127    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="util"
Oct 30 02:45:42.293146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293134    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="util"
Oct 30 02:45:42.293415 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293157    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="pull"
Oct 30 02:45:42.293415 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293165    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="pull"
Oct 30 02:45:42.293415 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.293264    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75caf35-e627-4b1f-a928-e3e8dbefdf15" containerName="extract"
Oct 30 02:45:42.295439 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.295418    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.326514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.326476    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 02:45:42.326514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.326493    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 02:45:42.326514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.326476    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ffrqx\""
Oct 30 02:45:42.327564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.327542    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"]
Oct 30 02:45:42.366729 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.366708    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.366833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.366737    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.366833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.366779    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4llt\" (UniqueName: \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.467500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.467464    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.467740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.467504    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.467740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.467566    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4llt\" (UniqueName: \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.467861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.467832    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.467914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.467865    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.486944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.486919    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4llt\" (UniqueName: \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.608821 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.608782    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:42.901139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:42.901113    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"]
Oct 30 02:45:42.903413 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:42.903388    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a40d11c_c5b7_4b2b_baae_4a13b587ab83.slice/crio-161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77 WatchSource:0}: Error finding container 161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77: Status 404 returned error can't find the container with id 161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77
Oct 30 02:45:43.103829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:43.103795    4214 generic.go:356] "Generic (PLEG): container finished" podID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerID="e621805e3aecdb8a453ced91a99199930fe273e75f90f8138ab990cac6884201" exitCode=0
Oct 30 02:45:43.104108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:43.103844    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv" event={"ID":"1a40d11c-c5b7-4b2b-baae-4a13b587ab83","Type":"ContainerDied","Data":"e621805e3aecdb8a453ced91a99199930fe273e75f90f8138ab990cac6884201"}
Oct 30 02:45:43.104108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:43.103893    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv" event={"ID":"1a40d11c-c5b7-4b2b-baae-4a13b587ab83","Type":"ContainerStarted","Data":"161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77"}
Oct 30 02:45:44.110447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:44.110417    4214 generic.go:356] "Generic (PLEG): container finished" podID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerID="f561212fdb49ec7be24802c2400c1548fd5eee3bce7dea20af0bdf46c0f8b637" exitCode=0
Oct 30 02:45:44.110825 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:44.110499    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv" event={"ID":"1a40d11c-c5b7-4b2b-baae-4a13b587ab83","Type":"ContainerDied","Data":"f561212fdb49ec7be24802c2400c1548fd5eee3bce7dea20af0bdf46c0f8b637"}
Oct 30 02:45:45.115918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:45.115885    4214 generic.go:356] "Generic (PLEG): container finished" podID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerID="ae418b026744bab6b0ceff231911833fdad3290e62874ea47c3b73c45a6d2dce" exitCode=0
Oct 30 02:45:45.116356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:45.115933    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv" event={"ID":"1a40d11c-c5b7-4b2b-baae-4a13b587ab83","Type":"ContainerDied","Data":"ae418b026744bab6b0ceff231911833fdad3290e62874ea47c3b73c45a6d2dce"}
Oct 30 02:45:46.238208 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.238184    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:46.298584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.298540    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\" (UniqueName: \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\") pod \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") "
Oct 30 02:45:46.298766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.298606    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\") pod \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") "
Oct 30 02:45:46.298766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.298664    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\") pod \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\" (UID: \"1a40d11c-c5b7-4b2b-baae-4a13b587ab83\") "
Oct 30 02:45:46.299260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.299207    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle" pod "1a40d11c-c5b7-4b2b-baae-4a13b587ab83" (UID: "1a40d11c-c5b7-4b2b-baae-4a13b587ab83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:46.300583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.300562    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt" pod "1a40d11c-c5b7-4b2b-baae-4a13b587ab83" (UID: "1a40d11c-c5b7-4b2b-baae-4a13b587ab83"). InnerVolumeSpecName "kube-api-access-h4llt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:45:46.304422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.304397    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util" pod "1a40d11c-c5b7-4b2b-baae-4a13b587ab83" (UID: "1a40d11c-c5b7-4b2b-baae-4a13b587ab83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:45:46.400133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.400063    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:46.400133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.400091    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4llt\" (UniqueName: \"kubernetes.io/projected/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-kube-api-access-h4llt\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:46.400133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:46.400103    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40d11c-c5b7-4b2b-baae-4a13b587ab83-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:45:47.125018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:47.124985    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv" event={"ID":"1a40d11c-c5b7-4b2b-baae-4a13b587ab83","Type":"ContainerDied","Data":"161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77"}
Oct 30 02:45:47.125161 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:47.125036    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161b2138b3d32fa159171073ef2e4af9c963301f0a1b1f625ba3e1bba3316e77"
Oct 30 02:45:47.125161 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:47.124998    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7146kdv"
Oct 30 02:45:52.485154 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485114    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"]
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485419    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="pull"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485430    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="pull"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485442    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="util"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485447    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="util"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485455    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="extract"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485460    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="extract"
Oct 30 02:45:52.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.485512    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a40d11c-c5b7-4b2b-baae-4a13b587ab83" containerName="extract"
Oct 30 02:45:52.488649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.488632    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.523031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.523008    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"]
Oct 30 02:45:52.542916 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.542890    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvcl\" (UniqueName: \"kubernetes.io/projected/ae0dbaef-117a-4285-aa15-47af049ee41e-kube-api-access-khvcl\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.543030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.542932    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.543030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.542951    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.543143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.543068    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.543143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.543096    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.543143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.543118    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.568320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568298    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 02:45:52.568581 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568491    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 02:45:52.568697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568682    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 02:45:52.568755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568735    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-knctd\""
Oct 30 02:45:52.568808 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568775    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 02:45:52.568808 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.568783    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 02:45:52.644153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644131    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644165    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644186    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644217    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvcl\" (UniqueName: \"kubernetes.io/projected/ae0dbaef-117a-4285-aa15-47af049ee41e-kube-api-access-khvcl\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644277    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644495 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644293    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.644670 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.644646    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.646534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.646508    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae0dbaef-117a-4285-aa15-47af049ee41e-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.646628 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.646594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.646687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.646654    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.646739 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.646708    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae0dbaef-117a-4285-aa15-47af049ee41e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.664507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.664482    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvcl\" (UniqueName: \"kubernetes.io/projected/ae0dbaef-117a-4285-aa15-47af049ee41e-kube-api-access-khvcl\") pod \"opendatahub-operator-controller-manager-75bcb9cf-b8rpz\" (UID: \"ae0dbaef-117a-4285-aa15-47af049ee41e\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:52.804596 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:52.804569    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:53.236152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:53.236058    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"]
Oct 30 02:45:53.252441 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:45:53.252413    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0dbaef_117a_4285_aa15_47af049ee41e.slice/crio-653590bd5817a8895fc14d833972a1a15bdd8d962ef1297f3aa773632546da34 WatchSource:0}: Error finding container 653590bd5817a8895fc14d833972a1a15bdd8d962ef1297f3aa773632546da34: Status 404 returned error can't find the container with id 653590bd5817a8895fc14d833972a1a15bdd8d962ef1297f3aa773632546da34
Oct 30 02:45:54.156206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:54.156156    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz" event={"ID":"ae0dbaef-117a-4285-aa15-47af049ee41e","Type":"ContainerStarted","Data":"653590bd5817a8895fc14d833972a1a15bdd8d962ef1297f3aa773632546da34"}
Oct 30 02:45:56.166303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:56.166267    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz" event={"ID":"ae0dbaef-117a-4285-aa15-47af049ee41e","Type":"ContainerStarted","Data":"69618222a99f04a9de0f19a3f74342eb1dea9243f1abb97a452961d7081df7b3"}
Oct 30 02:45:57.171150 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:57.171116    4214 generic.go:356] "Generic (PLEG): container finished" podID="ae0dbaef-117a-4285-aa15-47af049ee41e" containerID="69618222a99f04a9de0f19a3f74342eb1dea9243f1abb97a452961d7081df7b3" exitCode=0
Oct 30 02:45:57.171556 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:57.171211    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz" event={"ID":"ae0dbaef-117a-4285-aa15-47af049ee41e","Type":"ContainerDied","Data":"69618222a99f04a9de0f19a3f74342eb1dea9243f1abb97a452961d7081df7b3"}
Oct 30 02:45:58.177320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:58.177281    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz" event={"ID":"ae0dbaef-117a-4285-aa15-47af049ee41e","Type":"ContainerStarted","Data":"556421482ca3353bb600a8f0a18b462fb10693dd9242842ab27553e10306678d"}
Oct 30 02:45:58.177674 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:58.177436    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:45:58.259547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:45:58.259489    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz" podStartSLOduration=2.617698783 podStartE2EDuration="6.259471901s" totalImagesPullingTime="3.151151899s" totalInitContainerRuntime="490.621219ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:58.245208968 +0000 UTC m=+297.319379783" watchObservedRunningTime="2026-10-30 02:45:58.259471901 +0000 UTC m=+297.333642695"
Oct 30 02:46:09.162563 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:46:09.162519    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:46:09.182572 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:09.182547    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-b8rpz"
Oct 30 02:46:12.247416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.247384    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-j22s7"]
Oct 30 02:46:12.255883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.255860    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.271197 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.271172    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-j22s7"]
Oct 30 02:46:12.282389 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.282366    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 02:46:12.282639 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.282618    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 02:46:12.282705 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.282645    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fv4st\""
Oct 30 02:46:12.282814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.282799    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:46:12.282866 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.282856    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:46:12.394994 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.394958    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e261b600-8177-4830-9501-e26f627e6b8f-tmp\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.395150 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.395036    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlmc\" (UniqueName: \"kubernetes.io/projected/e261b600-8177-4830-9501-e26f627e6b8f-kube-api-access-mxlmc\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.395150 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.395074    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e261b600-8177-4830-9501-e26f627e6b8f-tls-certs\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.495825 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.495790    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e261b600-8177-4830-9501-e26f627e6b8f-tls-certs\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.495964 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.495842    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e261b600-8177-4830-9501-e26f627e6b8f-tmp\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.495964 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.495884    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlmc\" (UniqueName: \"kubernetes.io/projected/e261b600-8177-4830-9501-e26f627e6b8f-kube-api-access-mxlmc\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.498189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.498132    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e261b600-8177-4830-9501-e26f627e6b8f-tmp\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.498311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.498289    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e261b600-8177-4830-9501-e26f627e6b8f-tls-certs\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.516279 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.516256    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlmc\" (UniqueName: \"kubernetes.io/projected/e261b600-8177-4830-9501-e26f627e6b8f-kube-api-access-mxlmc\") pod \"kube-auth-proxy-64f884f994-j22s7\" (UID: \"e261b600-8177-4830-9501-e26f627e6b8f\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.567224 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.567204    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7"
Oct 30 02:46:12.860093 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.860067    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-j22s7"]
Oct 30 02:46:12.862750 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:46:12.862722    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode261b600_8177_4830_9501_e26f627e6b8f.slice/crio-c99586c19956de3f3f6d5130667fbae98ef8b88b7d92daa62c7d0a96a20db7a7 WatchSource:0}: Error finding container c99586c19956de3f3f6d5130667fbae98ef8b88b7d92daa62c7d0a96a20db7a7: Status 404 returned error can't find the container with id c99586c19956de3f3f6d5130667fbae98ef8b88b7d92daa62c7d0a96a20db7a7
Oct 30 02:46:12.864669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:12.864651    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:46:13.233896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.233808    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7" event={"ID":"e261b600-8177-4830-9501-e26f627e6b8f","Type":"ContainerStarted","Data":"c99586c19956de3f3f6d5130667fbae98ef8b88b7d92daa62c7d0a96a20db7a7"}
Oct 30 02:46:13.627704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.627672    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:46:13.631262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.631074    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.647796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.647774    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 02:46:13.647937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.647852    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ctkqx\""
Oct 30 02:46:13.647937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.647856    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 02:46:13.647937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.647774    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 02:46:13.662828 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.662797    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:46:13.675969 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.675945    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 02:46:13.705651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705615    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705785 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705701    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9vk\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705837 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705781    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705837 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705814    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705910 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705846    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705909    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.705946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.705938    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807037 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807006    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807070    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807096    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807139    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807165    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9vk\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807212    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807254    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.807928 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.807897    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.810578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.810553    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.810746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.810726    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.810815 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.810775    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.811773 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.811751    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.827782 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.827757    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.827885 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.827822    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9vk\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\") pod \"istiod-openshift-gateway-7876bfcb74-v4xpn\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:13.945949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:13.945869    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:14.267349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:14.267322    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:46:14.269593 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:46:14.269560    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5519e2fd_a2cc_4c4b_9c4a_c9fbd2a204fe.slice/crio-d4c84390c7ad2c9cf517c1ab7bc04650fb41b80183c40fe2a24c540cc4d15b94 WatchSource:0}: Error finding container d4c84390c7ad2c9cf517c1ab7bc04650fb41b80183c40fe2a24c540cc4d15b94: Status 404 returned error can't find the container with id d4c84390c7ad2c9cf517c1ab7bc04650fb41b80183c40fe2a24c540cc4d15b94
Oct 30 02:46:15.246793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:15.246727    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" event={"ID":"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe","Type":"ContainerStarted","Data":"d4c84390c7ad2c9cf517c1ab7bc04650fb41b80183c40fe2a24c540cc4d15b94"}
Oct 30 02:46:16.252369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:16.252331    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7" event={"ID":"e261b600-8177-4830-9501-e26f627e6b8f","Type":"ContainerStarted","Data":"05cd16c3c36bf104443df81320f4b8497cabc46079741d1ea6e1c075a6fb0125"}
Oct 30 02:46:16.298141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:16.298072    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64f884f994-j22s7" podStartSLOduration=1.831464709 podStartE2EDuration="4.298052528s" totalImagesPullingTime="2.466587819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:16.296581966 +0000 UTC m=+315.370752802" watchObservedRunningTime="2026-10-30 02:46:16.298052528 +0000 UTC m=+315.372223325"
Oct 30 02:46:16.603043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:16.603002    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:46:16.603172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:16.603083    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:46:17.265710 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:17.265663    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" event={"ID":"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe","Type":"ContainerStarted","Data":"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"}
Oct 30 02:46:17.336049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:17.335988    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" podStartSLOduration=2.005567406 podStartE2EDuration="4.335970858s" totalImagesPullingTime="2.330403452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:17.321730496 +0000 UTC m=+316.395901305" watchObservedRunningTime="2026-10-30 02:46:17.335970858 +0000 UTC m=+316.410141653"
Oct 30 02:46:18.270966 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:18.270928    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:18.272905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:18.272880    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:46:27.445801 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.445765    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"]
Oct 30 02:46:27.461121 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.461093    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.472750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.472728    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"]
Oct 30 02:46:27.484088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.484063    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8zd7t\""
Oct 30 02:46:27.510727 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.510703    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 02:46:27.634148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.634107    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fffz\" (UniqueName: \"kubernetes.io/projected/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-kube-api-access-9fffz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.634321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.634160    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.634321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.634274    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.634409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.634349    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.734756 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.734678    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.734756 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.734720    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fffz\" (UniqueName: \"kubernetes.io/projected/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-kube-api-access-9fffz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.734756 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.734747    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.735047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.734772    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.735137 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.735118    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.735199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.735118    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.735652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.735626    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.754858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.754831    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fffz\" (UniqueName: \"kubernetes.io/projected/bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9-kube-api-access-9fffz\") pod \"kserve-module-controller-manager-5cfb77bf8c-4zg7h\" (UID: \"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:27.776058 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:27.776028    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:28.109573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:28.109540    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"]
Oct 30 02:46:28.116077 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:46:28.116041    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2cad7e_ef2a_4091_b6bf_d3ceab7557f9.slice/crio-3467016339b461bb189ba5f5ab72f6612f72ef52b23f3af03052c6963286f45f WatchSource:0}: Error finding container 3467016339b461bb189ba5f5ab72f6612f72ef52b23f3af03052c6963286f45f: Status 404 returned error can't find the container with id 3467016339b461bb189ba5f5ab72f6612f72ef52b23f3af03052c6963286f45f
Oct 30 02:46:28.312311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:28.312279    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h" event={"ID":"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9","Type":"ContainerStarted","Data":"3467016339b461bb189ba5f5ab72f6612f72ef52b23f3af03052c6963286f45f"}
Oct 30 02:46:30.322821 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:30.322787    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h" event={"ID":"bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9","Type":"ContainerStarted","Data":"41644acc125aeef6d7943e8ef96c6554443abc25c7a967dc19fb9b0c22d99bae"}
Oct 30 02:46:30.323207 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:30.322863    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:39.163183 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:46:39.163142    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:46:51.347544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:51.347507    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h"
Oct 30 02:46:51.403387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:46:51.403335    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4zg7h" podStartSLOduration=22.658015978 podStartE2EDuration="24.403322147s" totalImagesPullingTime="1.745306169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:30.389024576 +0000 UTC m=+329.463195368" watchObservedRunningTime="2026-10-30 02:46:51.403322147 +0000 UTC m=+350.477492984"
Oct 30 02:47:09.163018 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:47:09.162967    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:47:12.475355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.475318    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"]
Oct 30 02:47:12.490323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.490298    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.523294 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.523263    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"]
Oct 30 02:47:12.585083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585050    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73fccb64-3ddb-4f4b-860c-b4953d677916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585100    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2pc\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-kube-api-access-2g2pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585121    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585212    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585364    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.585442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.585400    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686088    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686360 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686176    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73fccb64-3ddb-4f4b-860c-b4953d677916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686360 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686205    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2pc\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-kube-api-access-2g2pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686362    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686416    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686457    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.686977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.686951    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.688778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.688748    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.688875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.688776    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.688925 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.688904    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73fccb64-3ddb-4f4b-860c-b4953d677916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.689023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.689007    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.708561 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.708531    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.708654 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.708588    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2pc\" (UniqueName: \"kubernetes.io/projected/73fccb64-3ddb-4f4b-860c-b4953d677916-kube-api-access-2g2pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7cnx\" (UID: \"73fccb64-3ddb-4f4b-860c-b4953d677916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:12.802210 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:12.802184    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:13.113492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.113468    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"]
Oct 30 02:47:13.115479 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:47:13.115452    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fccb64_3ddb_4f4b_860c_b4953d677916.slice/crio-38037c807501b65ce4fff2e42260ca89c3bddf40876a09fd3c04b75687423d17 WatchSource:0}: Error finding container 38037c807501b65ce4fff2e42260ca89c3bddf40876a09fd3c04b75687423d17: Status 404 returned error can't find the container with id 38037c807501b65ce4fff2e42260ca89c3bddf40876a09fd3c04b75687423d17
Oct 30 02:47:13.117386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.117356    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:47:13.117489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.117434    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:47:13.506883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.506807    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx" event={"ID":"73fccb64-3ddb-4f4b-860c-b4953d677916","Type":"ContainerStarted","Data":"47e3335285dddaaf4203ef3cd5532111182d5523ed2408aba03bfcd9252e53ad"}
Oct 30 02:47:13.506883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.506863    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx" event={"ID":"73fccb64-3ddb-4f4b-860c-b4953d677916","Type":"ContainerStarted","Data":"38037c807501b65ce4fff2e42260ca89c3bddf40876a09fd3c04b75687423d17"}
Oct 30 02:47:13.507393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:13.506953    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:14.513026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.512996    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx"
Oct 30 02:47:14.571520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.571468    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7cnx" podStartSLOduration=2.571454244 podStartE2EDuration="2.571454244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:13.568005732 +0000 UTC m=+372.642176528" watchObservedRunningTime="2026-10-30 02:47:14.571454244 +0000 UTC m=+373.645625037"
Oct 30 02:47:14.602480 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.602443    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:47:14.602715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.602689    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" podUID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" containerName="discovery" containerID="cri-o://7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c" gracePeriod=30
Oct 30 02:47:14.842583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.842562    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:47:14.906619 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906591    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.906813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906639    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.906813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.906813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.906813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906756    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.906813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906801    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.907103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.906819    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\") pod \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\" (UID: \"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe\") "
Oct 30 02:47:14.907531 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.907468    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:47:14.909590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909560    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:47:14.909700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909600    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:47:14.909700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909638    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:47:14.909700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909691    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "kube-api-access-jf9vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:47:14.909872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909779    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:47:14.909872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:14.909799    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert" pod "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" (UID: "5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:47:15.007508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007475    4214 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-ca-configmap\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007501    4214 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-csr-dns-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007509    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9vk\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-kube-api-access-jf9vk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007711 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007523    4214 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-local-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007711 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007531    4214 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-cacerts\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007711 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007538    4214 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-kubeconfig\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.007711 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.007546    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:47:15.521671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.521582    4214 generic.go:356] "Generic (PLEG): container finished" podID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" containerID="7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c" exitCode=0
Oct 30 02:47:15.521671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.521642    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"
Oct 30 02:47:15.521671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.521658    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" event={"ID":"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe","Type":"ContainerDied","Data":"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"}
Oct 30 02:47:15.522178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.521733    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn" event={"ID":"5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe","Type":"ContainerDied","Data":"d4c84390c7ad2c9cf517c1ab7bc04650fb41b80183c40fe2a24c540cc4d15b94"}
Oct 30 02:47:15.522178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.521754    4214 scope.go:122] "RemoveContainer" containerID="7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"
Oct 30 02:47:15.529755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.529736    4214 scope.go:122] "RemoveContainer" containerID="7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"
Oct 30 02:47:15.530021 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:47:15.530002    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c\": container with ID starting with 7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c not found: ID does not exist" containerID="7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"
Oct 30 02:47:15.530084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.530029    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c"} err="failed to get container status \"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c\": rpc error: code = NotFound desc = could not find container \"7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c\": container with ID starting with 7ae48534f1dfca566fbfc491116e86975cc3483eaa4930a5dca48042f36c9e5c not found: ID does not exist"
Oct 30 02:47:15.587368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.587339    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:47:15.600021 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:15.599995    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v4xpn"]
Oct 30 02:47:17.064301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:47:17.064273    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" path="/var/lib/kubelet/pods/5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe/volumes"
Oct 30 02:47:39.163333 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:47:39.163293    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:48:07.283810 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.283777    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"]
Oct 30 02:48:07.284267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.284066    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" containerName="discovery"
Oct 30 02:48:07.284267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.284076    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" containerName="discovery"
Oct 30 02:48:07.284267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.284139    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5519e2fd-a2cc-4c4b-9c4a-c9fbd2a204fe" containerName="discovery"
Oct 30 02:48:07.288473 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.288453    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.305151 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.305124    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"]
Oct 30 02:48:07.316860 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.316841    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d6422\""
Oct 30 02:48:07.316980 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.316927    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 02:48:07.316980 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.316924    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 02:48:07.357735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.357705    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqjs\" (UniqueName: \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.357861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.357742    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.357861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.357820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.380188 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.380155    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"]
Oct 30 02:48:07.383601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.383582    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.402592 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.402558    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-76stp\""
Oct 30 02:48:07.404929 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.404908    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"]
Oct 30 02:48:07.458509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458483    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8mx\" (UniqueName: \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.458666 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458516    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqjs\" (UniqueName: \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.458666 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458538    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.458666 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458597    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.458833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458813    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.458948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458875    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.459000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.458942    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.459149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.459133    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.472734 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.472705    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"]
Oct 30 02:48:07.476090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.476076    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.482224 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.482199    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqjs\" (UniqueName: \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.494479 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.494452    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cf84m\""
Oct 30 02:48:07.496386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.496363    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"]
Oct 30 02:48:07.560152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560119    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.560342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560159    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.560342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560204    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qt2c\" (UniqueName: \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.560419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560341    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.560419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560371    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8mx\" (UniqueName: \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.560419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560406    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.560579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560562    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.560695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.560680    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.575990 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.575947    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"]
Oct 30 02:48:07.579543 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.579517    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.585413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.585387    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8mx\" (UniqueName: \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.596448 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.596417    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-j4rng\""
Oct 30 02:48:07.598707 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.598681    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"]
Oct 30 02:48:07.604639 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.604618    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:07.661804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661769    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvgs\" (UniqueName: \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.661926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661821    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.661926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661849    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qt2c\" (UniqueName: \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.661926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661885    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.661926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661905    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.661926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.661922    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.662318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.662297    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.662811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.662790    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.683085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.683052    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qt2c\" (UniqueName: \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.699683 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.699659    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:07.762812 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.762782    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.762962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.762828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.762962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.762889    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvgs\" (UniqueName: \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.763320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.763298    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.763398 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.763328    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.785352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.785323    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvgs\" (UniqueName: \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.800154 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.800132    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:07.903342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.903307    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:07.904219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.904121    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"]
Oct 30 02:48:07.906370 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:07.906343    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bed5da_f862_4cf6_8857_eb1c3fba7b41.slice/crio-2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c WatchSource:0}: Error finding container 2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c: Status 404 returned error can't find the container with id 2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c
Oct 30 02:48:07.996485 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:07.996453    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"]
Oct 30 02:48:07.999811 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:07.999784    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6837be30_fe0a_4a60_9c6a_fa7023db90f3.slice/crio-8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31 WatchSource:0}: Error finding container 8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31: Status 404 returned error can't find the container with id 8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31
Oct 30 02:48:08.096614 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.096587    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"]
Oct 30 02:48:08.098725 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:08.098695    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117af530_b927_430f_a06a_a49e5ce56e0c.slice/crio-71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81 WatchSource:0}: Error finding container 71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81: Status 404 returned error can't find the container with id 71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81
Oct 30 02:48:08.202288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.202262    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"]
Oct 30 02:48:08.206825 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:08.206797    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf32c97_d4ec_4e44_b61d_9a78380abb89.slice/crio-6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58 WatchSource:0}: Error finding container 6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58: Status 404 returned error can't find the container with id 6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58
Oct 30 02:48:08.732991 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.732901    4214 generic.go:356] "Generic (PLEG): container finished" podID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerID="559ce829adb1cd42ee9651e876f8a0c35e732a5bf9119d523e0deeb7a77bba80" exitCode=0
Oct 30 02:48:08.733464 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.732992    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f" event={"ID":"fcf32c97-d4ec-4e44-b61d-9a78380abb89","Type":"ContainerDied","Data":"559ce829adb1cd42ee9651e876f8a0c35e732a5bf9119d523e0deeb7a77bba80"}
Oct 30 02:48:08.733464 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.733052    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f" event={"ID":"fcf32c97-d4ec-4e44-b61d-9a78380abb89","Type":"ContainerStarted","Data":"6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58"}
Oct 30 02:48:08.734607 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.734575    4214 generic.go:356] "Generic (PLEG): container finished" podID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerID="0a2dd913ef4155206fb7d84824da8394d0ed6366f7c584878d0be41105394343" exitCode=0
Oct 30 02:48:08.734721 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.734664    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7" event={"ID":"6837be30-fe0a-4a60-9c6a-fa7023db90f3","Type":"ContainerDied","Data":"0a2dd913ef4155206fb7d84824da8394d0ed6366f7c584878d0be41105394343"}
Oct 30 02:48:08.734784 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.734720    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7" event={"ID":"6837be30-fe0a-4a60-9c6a-fa7023db90f3","Type":"ContainerStarted","Data":"8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31"}
Oct 30 02:48:08.736296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.736273    4214 generic.go:356] "Generic (PLEG): container finished" podID="117af530-b927-430f-a06a-a49e5ce56e0c" containerID="8068473def221199a845e3fb6f3c7a1b80043816a60240b78e686b49b528c58f" exitCode=0
Oct 30 02:48:08.736384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.736345    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn" event={"ID":"117af530-b927-430f-a06a-a49e5ce56e0c","Type":"ContainerDied","Data":"8068473def221199a845e3fb6f3c7a1b80043816a60240b78e686b49b528c58f"}
Oct 30 02:48:08.736384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.736381    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn" event={"ID":"117af530-b927-430f-a06a-a49e5ce56e0c","Type":"ContainerStarted","Data":"71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81"}
Oct 30 02:48:08.737940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.737921    4214 generic.go:356] "Generic (PLEG): container finished" podID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerID="1d5c0e502b2e987af397b39375508c12a3486fc8c1a88ef889315c879214bc8e" exitCode=0
Oct 30 02:48:08.738039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.737949    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf" event={"ID":"d5bed5da-f862-4cf6-8857-eb1c3fba7b41","Type":"ContainerDied","Data":"1d5c0e502b2e987af397b39375508c12a3486fc8c1a88ef889315c879214bc8e"}
Oct 30 02:48:08.738039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:08.737981    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf" event={"ID":"d5bed5da-f862-4cf6-8857-eb1c3fba7b41","Type":"ContainerStarted","Data":"2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c"}
Oct 30 02:48:09.170164 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:09.170121    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:48:09.750213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.750131    4214 generic.go:356] "Generic (PLEG): container finished" podID="117af530-b927-430f-a06a-a49e5ce56e0c" containerID="398c4a8324b26b3ff5fd3ad9882e6bfea4c2705f0a0385e660d9f9feaf7c2d83" exitCode=0
Oct 30 02:48:09.750213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.750191    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn" event={"ID":"117af530-b927-430f-a06a-a49e5ce56e0c","Type":"ContainerDied","Data":"398c4a8324b26b3ff5fd3ad9882e6bfea4c2705f0a0385e660d9f9feaf7c2d83"}
Oct 30 02:48:09.752779 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.752761    4214 generic.go:356] "Generic (PLEG): container finished" podID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerID="4309c98f47ed7b70ecd65831baf4b4b6be87a19e8266e887fea9993d11512db7" exitCode=0
Oct 30 02:48:09.752918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.752831    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf" event={"ID":"d5bed5da-f862-4cf6-8857-eb1c3fba7b41","Type":"ContainerDied","Data":"4309c98f47ed7b70ecd65831baf4b4b6be87a19e8266e887fea9993d11512db7"}
Oct 30 02:48:09.755189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.755169    4214 generic.go:356] "Generic (PLEG): container finished" podID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerID="b73371e4d1b622beb3e2273a910cbf34fe6c9d46b8b5aae4435d06957d5a4374" exitCode=0
Oct 30 02:48:09.755289 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.755265    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f" event={"ID":"fcf32c97-d4ec-4e44-b61d-9a78380abb89","Type":"ContainerDied","Data":"b73371e4d1b622beb3e2273a910cbf34fe6c9d46b8b5aae4435d06957d5a4374"}
Oct 30 02:48:09.756973 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.756951    4214 generic.go:356] "Generic (PLEG): container finished" podID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerID="2e3f0f26c263434ab5ab222a542964c84f08d03e56d1d12c4254e9e1cad16b9d" exitCode=0
Oct 30 02:48:09.757062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:09.757009    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7" event={"ID":"6837be30-fe0a-4a60-9c6a-fa7023db90f3","Type":"ContainerDied","Data":"2e3f0f26c263434ab5ab222a542964c84f08d03e56d1d12c4254e9e1cad16b9d"}
Oct 30 02:48:10.763751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.763679    4214 generic.go:356] "Generic (PLEG): container finished" podID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerID="c1a83f0f7019d77ee5744f81c4bb12e4b79b0c73ea8eeb4e35a7013b0b1a4842" exitCode=0
Oct 30 02:48:10.764142 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.763787    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf" event={"ID":"d5bed5da-f862-4cf6-8857-eb1c3fba7b41","Type":"ContainerDied","Data":"c1a83f0f7019d77ee5744f81c4bb12e4b79b0c73ea8eeb4e35a7013b0b1a4842"}
Oct 30 02:48:10.765962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.765938    4214 generic.go:356] "Generic (PLEG): container finished" podID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerID="5d491c4cc6c4b84904d6a8c1d2c985b76d8bc53099762b8f6b1974c1e43fd472" exitCode=0
Oct 30 02:48:10.766084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.766016    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f" event={"ID":"fcf32c97-d4ec-4e44-b61d-9a78380abb89","Type":"ContainerDied","Data":"5d491c4cc6c4b84904d6a8c1d2c985b76d8bc53099762b8f6b1974c1e43fd472"}
Oct 30 02:48:10.768251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.768214    4214 generic.go:356] "Generic (PLEG): container finished" podID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerID="af641e2ccdf032fa7d7133016117d72b02cd6d1b0ee34311fc8f86aca5a6364f" exitCode=0
Oct 30 02:48:10.768363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.768313    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7" event={"ID":"6837be30-fe0a-4a60-9c6a-fa7023db90f3","Type":"ContainerDied","Data":"af641e2ccdf032fa7d7133016117d72b02cd6d1b0ee34311fc8f86aca5a6364f"}
Oct 30 02:48:10.770432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.770411    4214 generic.go:356] "Generic (PLEG): container finished" podID="117af530-b927-430f-a06a-a49e5ce56e0c" containerID="cc4e525bc7a74850866e3d2c0aaaf7bf8dc8d24fb5200f3e22e20e8e0739c64b" exitCode=0
Oct 30 02:48:10.770547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:10.770438    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn" event={"ID":"117af530-b927-430f-a06a-a49e5ce56e0c","Type":"ContainerDied","Data":"cc4e525bc7a74850866e3d2c0aaaf7bf8dc8d24fb5200f3e22e20e8e0739c64b"}
Oct 30 02:48:11.909611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:11.909585    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:11.950339 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:11.950298    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:11.966027 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:11.966006    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:11.971775 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:11.971752    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:12.005550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.005520    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\") pod \"117af530-b927-430f-a06a-a49e5ce56e0c\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") "
Oct 30 02:48:12.005725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.005661    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\" (UniqueName: \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\") pod \"117af530-b927-430f-a06a-a49e5ce56e0c\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") "
Oct 30 02:48:12.005725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.005716    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\") pod \"117af530-b927-430f-a06a-a49e5ce56e0c\" (UID: \"117af530-b927-430f-a06a-a49e5ce56e0c\") "
Oct 30 02:48:12.006256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.006206    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle" pod "117af530-b927-430f-a06a-a49e5ce56e0c" (UID: "117af530-b927-430f-a06a-a49e5ce56e0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.013344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.013308    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util" pod "117af530-b927-430f-a06a-a49e5ce56e0c" (UID: "117af530-b927-430f-a06a-a49e5ce56e0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.014036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.014011    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c" pod "117af530-b927-430f-a06a-a49e5ce56e0c" (UID: "117af530-b927-430f-a06a-a49e5ce56e0c"). InnerVolumeSpecName "kube-api-access-2qt2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:12.106293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106264    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\") pod \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") "
Oct 30 02:48:12.106489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106316    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\" (UniqueName: \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\") pod \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") "
Oct 30 02:48:12.106489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106357    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\" (UniqueName: \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\") pod \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") "
Oct 30 02:48:12.106489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106387    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\") pod \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") "
Oct 30 02:48:12.106489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106431    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\") pod \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") "
Oct 30 02:48:12.106489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106479    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\") pod \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\" (UID: \"fcf32c97-d4ec-4e44-b61d-9a78380abb89\") "
Oct 30 02:48:12.106740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106522    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\" (UniqueName: \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\") pod \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") "
Oct 30 02:48:12.106740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106565    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\") pod \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\" (UID: \"d5bed5da-f862-4cf6-8857-eb1c3fba7b41\") "
Oct 30 02:48:12.106740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106634    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\") pod \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\" (UID: \"6837be30-fe0a-4a60-9c6a-fa7023db90f3\") "
Oct 30 02:48:12.106887 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106868    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.106949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106891    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/117af530-b927-430f-a06a-a49e5ce56e0c-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.106949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.106906    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qt2c\" (UniqueName: \"kubernetes.io/projected/117af530-b927-430f-a06a-a49e5ce56e0c-kube-api-access-2qt2c\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.107482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.107362    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle" pod "6837be30-fe0a-4a60-9c6a-fa7023db90f3" (UID: "6837be30-fe0a-4a60-9c6a-fa7023db90f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.107482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.107436    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle" pod "fcf32c97-d4ec-4e44-b61d-9a78380abb89" (UID: "fcf32c97-d4ec-4e44-b61d-9a78380abb89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.108054 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.108011    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle" pod "d5bed5da-f862-4cf6-8857-eb1c3fba7b41" (UID: "d5bed5da-f862-4cf6-8857-eb1c3fba7b41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.108855 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.108823    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs" pod "d5bed5da-f862-4cf6-8857-eb1c3fba7b41" (UID: "d5bed5da-f862-4cf6-8857-eb1c3fba7b41"). InnerVolumeSpecName "kube-api-access-5lqjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:12.108946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.108926    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs" pod "fcf32c97-d4ec-4e44-b61d-9a78380abb89" (UID: "fcf32c97-d4ec-4e44-b61d-9a78380abb89"). InnerVolumeSpecName "kube-api-access-lnvgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:12.109364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.109341    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx" pod "6837be30-fe0a-4a60-9c6a-fa7023db90f3" (UID: "6837be30-fe0a-4a60-9c6a-fa7023db90f3"). InnerVolumeSpecName "kube-api-access-hs8mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:12.113202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.113169    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util" pod "6837be30-fe0a-4a60-9c6a-fa7023db90f3" (UID: "6837be30-fe0a-4a60-9c6a-fa7023db90f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.114697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.114674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util" pod "fcf32c97-d4ec-4e44-b61d-9a78380abb89" (UID: "fcf32c97-d4ec-4e44-b61d-9a78380abb89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.115467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.115446    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util" pod "d5bed5da-f862-4cf6-8857-eb1c3fba7b41" (UID: "d5bed5da-f862-4cf6-8857-eb1c3fba7b41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:12.208153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208116    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvgs\" (UniqueName: \"kubernetes.io/projected/fcf32c97-d4ec-4e44-b61d-9a78380abb89-kube-api-access-lnvgs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208150    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208165    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208177    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208188    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8mx\" (UniqueName: \"kubernetes.io/projected/6837be30-fe0a-4a60-9c6a-fa7023db90f3-kube-api-access-hs8mx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208200    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208210    4214 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6837be30-fe0a-4a60-9c6a-fa7023db90f3-bundle\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208220    4214 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcf32c97-d4ec-4e44-b61d-9a78380abb89-util\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.208390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.208256    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lqjs\" (UniqueName: \"kubernetes.io/projected/d5bed5da-f862-4cf6-8857-eb1c3fba7b41-kube-api-access-5lqjs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:12.781200 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.781164    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f" event={"ID":"fcf32c97-d4ec-4e44-b61d-9a78380abb89","Type":"ContainerDied","Data":"6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58"}
Oct 30 02:48:12.781435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.781216    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f2165cf70f39ee77831706759876f0fa9ff3abb84b0ef38235d1f3ae529ba58"
Oct 30 02:48:12.781435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.781178    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awgb2f"
Oct 30 02:48:12.783124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.783103    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7"
Oct 30 02:48:12.783248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.783139    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ck6n7" event={"ID":"6837be30-fe0a-4a60-9c6a-fa7023db90f3","Type":"ContainerDied","Data":"8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31"}
Oct 30 02:48:12.783248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.783185    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dbdb98ae4b2b20b72475068167bb9df17120a48c488b7f0521a0ead9a9b7d31"
Oct 30 02:48:12.785125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.785098    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn"
Oct 30 02:48:12.785256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.785130    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crrtbn" event={"ID":"117af530-b927-430f-a06a-a49e5ce56e0c","Type":"ContainerDied","Data":"71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81"}
Oct 30 02:48:12.785256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.785169    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71bd196113180cd39f609851baf1bb053a4042231f7e7f27541a7acea6d41a81"
Oct 30 02:48:12.787111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.787085    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf" event={"ID":"d5bed5da-f862-4cf6-8857-eb1c3fba7b41","Type":"ContainerDied","Data":"2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c"}
Oct 30 02:48:12.787189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.787128    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de7b47e06172c70eeba53fa06eee66d7132525cb567afcc272c88b8671b920c"
Oct 30 02:48:12.787189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:12.787151    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67z8jf"
Oct 30 02:48:17.817347 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817319    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"]
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817596    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817606    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817616    4214 cpu_manager.go:414] "containerMap: removing container" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="extract"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817621    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="extract"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817629    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817633    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817640    4214 cpu_manager.go:414] "containerMap: removing container" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817644    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817650    4214 cpu_manager.go:414] "containerMap: removing container" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817654    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817659    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="extract"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817664    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="extract"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817671    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817675    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="util"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817681    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817685    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817696    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817700    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="pull"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817715    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="extract"
Oct 30 02:48:17.817714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817719    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817726    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="pull"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817731    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="pull"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817738    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817742    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817787    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bed5da-f862-4cf6-8857-eb1c3fba7b41" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817795    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6837be30-fe0a-4a60-9c6a-fa7023db90f3" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817802    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="117af530-b927-430f-a06a-a49e5ce56e0c" containerName="extract"
Oct 30 02:48:17.818445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.817808    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf32c97-d4ec-4e44-b61d-9a78380abb89" containerName="extract"
Oct 30 02:48:17.821068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.821052    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:17.838589 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.838564    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 02:48:17.838709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.838586    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 02:48:17.852370 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.852343    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-km2wv\""
Oct 30 02:48:17.854213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.854188    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"]
Oct 30 02:48:17.965611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:17.965578    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fnt\" (UniqueName: \"kubernetes.io/projected/098ffa02-d582-4f82-96bc-3767ef975589-kube-api-access-58fnt\") pod \"authorino-operator-68d6dd76dc-d6ndt\" (UID: \"098ffa02-d582-4f82-96bc-3767ef975589\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:18.066337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:18.066298    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fnt\" (UniqueName: \"kubernetes.io/projected/098ffa02-d582-4f82-96bc-3767ef975589-kube-api-access-58fnt\") pod \"authorino-operator-68d6dd76dc-d6ndt\" (UID: \"098ffa02-d582-4f82-96bc-3767ef975589\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:18.092719 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:18.092653    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fnt\" (UniqueName: \"kubernetes.io/projected/098ffa02-d582-4f82-96bc-3767ef975589-kube-api-access-58fnt\") pod \"authorino-operator-68d6dd76dc-d6ndt\" (UID: \"098ffa02-d582-4f82-96bc-3767ef975589\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:18.133571 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:18.133545    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:18.463080 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:18.463047    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098ffa02_d582_4f82_96bc_3767ef975589.slice/crio-861a71e8735e48e6e7309c9c155e7e147ab8934a47d08ae2b8ede19d5c6fb06c WatchSource:0}: Error finding container 861a71e8735e48e6e7309c9c155e7e147ab8934a47d08ae2b8ede19d5c6fb06c: Status 404 returned error can't find the container with id 861a71e8735e48e6e7309c9c155e7e147ab8934a47d08ae2b8ede19d5c6fb06c
Oct 30 02:48:18.463196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:18.463088    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"]
Oct 30 02:48:18.817824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:18.817774    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt" event={"ID":"098ffa02-d582-4f82-96bc-3767ef975589","Type":"ContainerStarted","Data":"861a71e8735e48e6e7309c9c155e7e147ab8934a47d08ae2b8ede19d5c6fb06c"}
Oct 30 02:48:20.829571 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:20.829539    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt" event={"ID":"098ffa02-d582-4f82-96bc-3767ef975589","Type":"ContainerStarted","Data":"b0bf6b6e59fb72cc9d0170265d115c9be2fcf715785059c0ca2e1aa655d2ee02"}
Oct 30 02:48:20.830056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:20.829633    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:20.869742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:20.869693    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt" podStartSLOduration=1.846817382 podStartE2EDuration="3.869679046s" totalImagesPullingTime="2.022861664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:20.868024731 +0000 UTC m=+439.942195526" watchObservedRunningTime="2026-10-30 02:48:20.869679046 +0000 UTC m=+439.943849839"
Oct 30 02:48:22.152031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.151993    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"]
Oct 30 02:48:22.155428 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.155409    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.173115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.173088    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"]
Oct 30 02:48:22.184060 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.184035    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g7ggq\""
Oct 30 02:48:22.200369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.200343    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wd44\" (UniqueName: \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.200462 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.200419    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.301292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.301260    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.301460 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.301319    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wd44\" (UniqueName: \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.301668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.301644    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.324791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.324760    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wd44\" (UniqueName: \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.468044 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.467936    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:22.795454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.795300    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"]
Oct 30 02:48:22.798043 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:22.798015    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb09e30d_302e_4996_8408_4c4de9ee12fb.slice/crio-704185e42f9226472b60973f16e9a19e2f4db5dd0a7b69c0a0a1b45aceba39e8 WatchSource:0}: Error finding container 704185e42f9226472b60973f16e9a19e2f4db5dd0a7b69c0a0a1b45aceba39e8: Status 404 returned error can't find the container with id 704185e42f9226472b60973f16e9a19e2f4db5dd0a7b69c0a0a1b45aceba39e8
Oct 30 02:48:22.839471 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:22.839441    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v" event={"ID":"bb09e30d-302e-4996-8408-4c4de9ee12fb","Type":"ContainerStarted","Data":"704185e42f9226472b60973f16e9a19e2f4db5dd0a7b69c0a0a1b45aceba39e8"}
Oct 30 02:48:26.614758 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.614720    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"]
Oct 30 02:48:26.621081 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.621059    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:26.634238 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.634202    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hs2\" (UniqueName: \"kubernetes.io/projected/4288f6cc-f324-4098-a8ab-a59d418fcc2b-kube-api-access-b8hs2\") pod \"dns-operator-controller-manager-65bfd79598-qqpht\" (UID: \"4288f6cc-f324-4098-a8ab-a59d418fcc2b\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:26.638446 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.638380    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 02:48:26.651442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.651416    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zlfnd\""
Oct 30 02:48:26.653365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.653341    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"]
Oct 30 02:48:26.735284 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.735249    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hs2\" (UniqueName: \"kubernetes.io/projected/4288f6cc-f324-4098-a8ab-a59d418fcc2b-kube-api-access-b8hs2\") pod \"dns-operator-controller-manager-65bfd79598-qqpht\" (UID: \"4288f6cc-f324-4098-a8ab-a59d418fcc2b\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:26.763841 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.763816    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hs2\" (UniqueName: \"kubernetes.io/projected/4288f6cc-f324-4098-a8ab-a59d418fcc2b-kube-api-access-b8hs2\") pod \"dns-operator-controller-manager-65bfd79598-qqpht\" (UID: \"4288f6cc-f324-4098-a8ab-a59d418fcc2b\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:26.863812 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.863776    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v" event={"ID":"bb09e30d-302e-4996-8408-4c4de9ee12fb","Type":"ContainerStarted","Data":"3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf"}
Oct 30 02:48:26.863982 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.863880    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:26.910730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.910651    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v" podStartSLOduration=1.5905936889999999 podStartE2EDuration="4.910636407s" totalImagesPullingTime="3.320042718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:26.909362635 +0000 UTC m=+445.983533427" watchObservedRunningTime="2026-10-30 02:48:26.910636407 +0000 UTC m=+445.984807200"
Oct 30 02:48:26.932858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:26.932833    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:27.231164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:27.231081    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"]
Oct 30 02:48:27.234114 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:27.234089    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4288f6cc_f324_4098_a8ab_a59d418fcc2b.slice/crio-334dced34542bdda0589ad242af5d2d0c8bf357b568494472865804cd7a71333 WatchSource:0}: Error finding container 334dced34542bdda0589ad242af5d2d0c8bf357b568494472865804cd7a71333: Status 404 returned error can't find the container with id 334dced34542bdda0589ad242af5d2d0c8bf357b568494472865804cd7a71333
Oct 30 02:48:27.870702 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:27.870662    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht" event={"ID":"4288f6cc-f324-4098-a8ab-a59d418fcc2b","Type":"ContainerStarted","Data":"334dced34542bdda0589ad242af5d2d0c8bf357b568494472865804cd7a71333"}
Oct 30 02:48:28.877362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:28.877330    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht" event={"ID":"4288f6cc-f324-4098-a8ab-a59d418fcc2b","Type":"ContainerStarted","Data":"c0bc9a83456ba9250cad5857acaeb643770df7d77952c474696b3295ad99be4a"}
Oct 30 02:48:28.877805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:28.877419    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:28.919293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:28.919245    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht" podStartSLOduration=1.77042429 podStartE2EDuration="2.919216866s" totalImagesPullingTime="1.148792576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:28.917865333 +0000 UTC m=+447.992036127" watchObservedRunningTime="2026-10-30 02:48:28.919216866 +0000 UTC m=+447.993387658"
Oct 30 02:48:31.836606 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:31.836577    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-d6ndt"
Oct 30 02:48:37.873024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:37.872996    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:39.162064 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:39.162018    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:48:39.564420 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.564384    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:39.567922 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.567904    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.604276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.604247    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:39.640151 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.640124    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.640347 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.640175    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pw2\" (UniqueName: \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.741435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.741397    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.741611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.741450    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pw2\" (UniqueName: \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.741779 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.741760    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.770112 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.770082    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pw2\" (UniqueName: \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\") pod \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.772761 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.772739    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:39.773029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.773013    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:39.884816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:39.884786    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-qqpht"
Oct 30 02:48:40.085948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.085921    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:40.087889 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:40.087864    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41241cc_2c31_4871_a2ee_fb7e0f6e75dd.slice/crio-19047c5434b28c67f0a7bea9f1d62eb66a673877f1645967752b887d52629226 WatchSource:0}: Error finding container 19047c5434b28c67f0a7bea9f1d62eb66a673877f1645967752b887d52629226: Status 404 returned error can't find the container with id 19047c5434b28c67f0a7bea9f1d62eb66a673877f1645967752b887d52629226
Oct 30 02:48:40.934430 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.934392    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7" event={"ID":"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd","Type":"ContainerStarted","Data":"6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a"}
Oct 30 02:48:40.934944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.934455    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7" event={"ID":"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd","Type":"ContainerStarted","Data":"19047c5434b28c67f0a7bea9f1d62eb66a673877f1645967752b887d52629226"}
Oct 30 02:48:40.934944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.934488    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" containerName="manager" containerID="cri-o://6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a" gracePeriod=10
Oct 30 02:48:40.934944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.934651    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:40.992039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:40.991992    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7" podStartSLOduration=1.991977947 podStartE2EDuration="1.991977947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:40.977482054 +0000 UTC m=+460.051652847" watchObservedRunningTime="2026-10-30 02:48:40.991977947 +0000 UTC m=+460.066148739"
Oct 30 02:48:41.182091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.182067    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:41.253950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.253867    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\") pod \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") "
Oct 30 02:48:41.253950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.253903    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\" (UniqueName: \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\") pod \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\" (UID: \"d41241cc-2c31-4871-a2ee-fb7e0f6e75dd\") "
Oct 30 02:48:41.254418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.254392    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume" pod "d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" (UID: "d41241cc-2c31-4871-a2ee-fb7e0f6e75dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:41.256081 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.256055    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2" pod "d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" (UID: "d41241cc-2c31-4871-a2ee-fb7e0f6e75dd"). InnerVolumeSpecName "kube-api-access-x4pw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:41.355175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.355143    4214 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-extensions-socket-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:41.355175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.355170    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pw2\" (UniqueName: \"kubernetes.io/projected/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd-kube-api-access-x4pw2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:41.436430 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.436394    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"]
Oct 30 02:48:41.436621 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.436585    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" containerName="manager" containerID="cri-o://3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf" gracePeriod=2
Oct 30 02:48:41.441101 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.441072    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"]
Oct 30 02:48:41.450847 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.450818    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.464133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.464109    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:41.468918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.468893    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:41.469426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469409    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" containerName="manager"
Oct 30 02:48:41.469595 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469429    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" containerName="manager"
Oct 30 02:48:41.469595 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469526    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" containerName="manager"
Oct 30 02:48:41.469595 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469535    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" containerName="manager"
Oct 30 02:48:41.469736 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469608    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" containerName="manager"
Oct 30 02:48:41.469736 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.469620    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" containerName="manager"
Oct 30 02:48:41.472811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.472791    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.473362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.473344    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"]
Oct 30 02:48:41.499219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.499192    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:41.537919 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.537896    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:48:41.541035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.541012    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.548658 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.548627    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.556604 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.556585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.556663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.556620    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztjn\" (UniqueName: \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.574242 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.574201    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:48:41.614375 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.614338    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.657598 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.657566    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.657735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.657625    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.657735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.657695    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cdd\" (UniqueName: \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.657810 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.657762    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztjn\" (UniqueName: \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.658024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.658006    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.676861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.676837    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.696567 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.696532    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.702133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.702117    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztjn\" (UniqueName: \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\") pod \"kuadrant-operator-controller-manager-6b6fb56846-466sm\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.758105 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758080    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\") pod \"bb09e30d-302e-4996-8408-4c4de9ee12fb\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") "
Oct 30 02:48:41.758223 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758122    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\" (UniqueName: \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\") pod \"bb09e30d-302e-4996-8408-4c4de9ee12fb\" (UID: \"bb09e30d-302e-4996-8408-4c4de9ee12fb\") "
Oct 30 02:48:41.758313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758248    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.758371 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758306    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cdd\" (UniqueName: \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.758530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758500    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume" pod "bb09e30d-302e-4996-8408-4c4de9ee12fb" (UID: "bb09e30d-302e-4996-8408-4c4de9ee12fb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:41.758656 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.758578    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.760061 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.760042    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44" pod "bb09e30d-302e-4996-8408-4c4de9ee12fb" (UID: "bb09e30d-302e-4996-8408-4c4de9ee12fb"). InnerVolumeSpecName "kube-api-access-5wd44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:41.781643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.781573    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cdd\" (UniqueName: \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\") pod \"kuadrant-operator-controller-manager-956956bc4-b9n6m\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.830224 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.830197    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:41.854102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.854078    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:41.859153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.859132    4214 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb09e30d-302e-4996-8408-4c4de9ee12fb-extensions-socket-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:41.859269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.859156    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wd44\" (UniqueName: \"kubernetes.io/projected/bb09e30d-302e-4996-8408-4c4de9ee12fb-kube-api-access-5wd44\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:41.940659 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.940626    4214 generic.go:356] "Generic (PLEG): container finished" podID="bb09e30d-302e-4996-8408-4c4de9ee12fb" containerID="3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf" exitCode=0
Oct 30 02:48:41.941071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.940689    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.941071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.940725    4214 scope.go:122] "RemoveContainer" containerID="3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf"
Oct 30 02:48:41.942102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.942056    4214 generic.go:356] "Generic (PLEG): container finished" podID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" containerID="6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a" exitCode=0
Oct 30 02:48:41.942268 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.942105    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:41.955437 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.955149    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.964902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.964884    4214 scope.go:122] "RemoveContainer" containerID="3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf"
Oct 30 02:48:41.965193 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.965175    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf\": container with ID starting with 3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf not found: ID does not exist" containerID="3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf"
Oct 30 02:48:41.965260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.965202    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf"} err="failed to get container status \"3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf\": rpc error: code = NotFound desc = could not find container \"3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf\": container with ID starting with 3b3931ecc211dda7ea9956f997421312049382541c1aaecf57c64f76dd048adf not found: ID does not exist"
Oct 30 02:48:41.965260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.965217    4214 scope.go:122] "RemoveContainer" containerID="6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a"
Oct 30 02:48:41.969350 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.969323    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:41.973444 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.973422    4214 scope.go:122] "RemoveContainer" containerID="6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a"
Oct 30 02:48:41.973774 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.973743    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a\": container with ID starting with 6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a not found: ID does not exist" containerID="6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a"
Oct 30 02:48:41.973861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:41.973786    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a"} err="failed to get container status \"6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a\": rpc error: code = NotFound desc = could not find container \"6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a\": container with ID starting with 6b9b86e38ece077517a0cabc8e07ca9a81dc787b06d16d4f6b2bfd970742db6a not found: ID does not exist"
Oct 30 02:48:41.983375 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.983346    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:41.997393 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:41.997362    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:42.011423 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:42.011402    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:42.144915 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.144882    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:42.148423 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:42.148394    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce30dee_4ca7_4fbf_992a_27538b10f933.slice/crio-4a67cbdb4eb5d366ab07db32a9fa3dd534e40fc95239845fb730b8e0eb1d0bc9 WatchSource:0}: Error finding container 4a67cbdb4eb5d366ab07db32a9fa3dd534e40fc95239845fb730b8e0eb1d0bc9: Status 404 returned error can't find the container with id 4a67cbdb4eb5d366ab07db32a9fa3dd534e40fc95239845fb730b8e0eb1d0bc9
Oct 30 02:48:42.163933 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.163915    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:48:42.166612 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:42.166582    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a7f0a9_9a2d_4af7_b6d0_51e5053700e4.slice/crio-318596c97b9d5e210876bd4dbeef00bbea7c36c8982820ecc3326d4f6459adbd WatchSource:0}: Error finding container 318596c97b9d5e210876bd4dbeef00bbea7c36c8982820ecc3326d4f6459adbd: Status 404 returned error can't find the container with id 318596c97b9d5e210876bd4dbeef00bbea7c36c8982820ecc3326d4f6459adbd
Oct 30 02:48:42.950678 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.950644    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" event={"ID":"bce30dee-4ca7-4fbf-992a-27538b10f933","Type":"ContainerStarted","Data":"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"}
Oct 30 02:48:42.951122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.950700    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" event={"ID":"bce30dee-4ca7-4fbf-992a-27538b10f933","Type":"ContainerStarted","Data":"4a67cbdb4eb5d366ab07db32a9fa3dd534e40fc95239845fb730b8e0eb1d0bc9"}
Oct 30 02:48:42.951122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.950727    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:42.952344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.952321    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" event={"ID":"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4","Type":"ContainerStarted","Data":"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"}
Oct 30 02:48:42.952440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.952363    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" event={"ID":"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4","Type":"ContainerStarted","Data":"318596c97b9d5e210876bd4dbeef00bbea7c36c8982820ecc3326d4f6459adbd"}
Oct 30 02:48:42.952440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.952396    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:42.996004 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:42.995956    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" podStartSLOduration=1.995943296 podStartE2EDuration="1.995943296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:42.994923745 +0000 UTC m=+462.069094587" watchObservedRunningTime="2026-10-30 02:48:42.995943296 +0000 UTC m=+462.070114139"
Oct 30 02:48:43.009972 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:43.009948    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:43.024008 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:43.023982    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:43.038028 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:43.037999    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-b9kw7\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-b9kw7"
Oct 30 02:48:43.064868 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:43.064839    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" path="/var/lib/kubelet/pods/bb09e30d-302e-4996-8408-4c4de9ee12fb/volumes"
Oct 30 02:48:43.065295 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:43.065264    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41241cc-2c31-4871-a2ee-fb7e0f6e75dd" path="/var/lib/kubelet/pods/d41241cc-2c31-4871-a2ee-fb7e0f6e75dd/volumes"
Oct 30 02:48:43.085640 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:43.085591    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" podStartSLOduration=2.085579328 podStartE2EDuration="2.085579328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:43.084383158 +0000 UTC m=+462.158553952" watchObservedRunningTime="2026-10-30 02:48:43.085579328 +0000 UTC m=+462.159750120"
Oct 30 02:48:43.098362 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:43.098335    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xqt2v\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="bb09e30d-302e-4996-8408-4c4de9ee12fb" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xqt2v"
Oct 30 02:48:53.959196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:53.959161    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:53.959598 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:53.959366    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:48:54.100704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.100675    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:54.100910 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.100887    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" podUID="bce30dee-4ca7-4fbf-992a-27538b10f933" containerName="manager" containerID="cri-o://8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd" gracePeriod=10
Oct 30 02:48:54.358252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.358210    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:54.422853 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.422787    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 02:48:54.423275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.423256    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bce30dee-4ca7-4fbf-992a-27538b10f933" containerName="manager"
Oct 30 02:48:54.423335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.423277    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce30dee-4ca7-4fbf-992a-27538b10f933" containerName="manager"
Oct 30 02:48:54.423374 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.423349    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce30dee-4ca7-4fbf-992a-27538b10f933" containerName="manager"
Oct 30 02:48:54.426609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.426592    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.448819 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.448788    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 02:48:54.465851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.465819    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\") pod \"bce30dee-4ca7-4fbf-992a-27538b10f933\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") "
Oct 30 02:48:54.466023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.465895    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\" (UniqueName: \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\") pod \"bce30dee-4ca7-4fbf-992a-27538b10f933\" (UID: \"bce30dee-4ca7-4fbf-992a-27538b10f933\") "
Oct 30 02:48:54.466348 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.466320    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume" pod "bce30dee-4ca7-4fbf-992a-27538b10f933" (UID: "bce30dee-4ca7-4fbf-992a-27538b10f933"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:48:54.467968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.467944    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn" pod "bce30dee-4ca7-4fbf-992a-27538b10f933" (UID: "bce30dee-4ca7-4fbf-992a-27538b10f933"). InnerVolumeSpecName "kube-api-access-5ztjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:48:54.566521 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.566479    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4td\" (UniqueName: \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.566693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.566598    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.566693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.566676    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztjn\" (UniqueName: \"kubernetes.io/projected/bce30dee-4ca7-4fbf-992a-27538b10f933-kube-api-access-5ztjn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:54.566693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.566691    4214 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bce30dee-4ca7-4fbf-992a-27538b10f933-extensions-socket-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:48:54.668184 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.668140    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4td\" (UniqueName: \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.668381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.668255    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.668613 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.668594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.688342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.688310    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4td\" (UniqueName: \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\") pod \"kuadrant-operator-controller-manager-57bb9bfff-44dk7\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:54.740111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:54.740073    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:55.010544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.010473    4214 generic.go:356] "Generic (PLEG): container finished" podID="bce30dee-4ca7-4fbf-992a-27538b10f933" containerID="8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd" exitCode=0
Oct 30 02:48:55.010544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.010523    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"
Oct 30 02:48:55.010544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.010538    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" event={"ID":"bce30dee-4ca7-4fbf-992a-27538b10f933","Type":"ContainerDied","Data":"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"}
Oct 30 02:48:55.010996 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.010581    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm" event={"ID":"bce30dee-4ca7-4fbf-992a-27538b10f933","Type":"ContainerDied","Data":"4a67cbdb4eb5d366ab07db32a9fa3dd534e40fc95239845fb730b8e0eb1d0bc9"}
Oct 30 02:48:55.010996 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.010603    4214 scope.go:122] "RemoveContainer" containerID="8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"
Oct 30 02:48:55.019032 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.019015    4214 scope.go:122] "RemoveContainer" containerID="8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"
Oct 30 02:48:55.019293 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:48:55.019277    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd\": container with ID starting with 8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd not found: ID does not exist" containerID="8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"
Oct 30 02:48:55.019356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.019302    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd"} err="failed to get container status \"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd\": rpc error: code = NotFound desc = could not find container \"8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd\": container with ID starting with 8530cdbb3aa9153e50e308595665ad75056312fa2e2ae429102bad96a9872cfd not found: ID does not exist"
Oct 30 02:48:55.066963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.066938    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:55.069903 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.069882    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 02:48:55.080325 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:55.080298    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-466sm"]
Oct 30 02:48:55.083764 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:48:55.083731    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6627dcdc_059c_41ba_b173_f91031ad4db1.slice/crio-4a739e459ed354cfdca54cd4d0d96c1c7e1e4a8a61ce6cf3ab11949f1a66d41e WatchSource:0}: Error finding container 4a739e459ed354cfdca54cd4d0d96c1c7e1e4a8a61ce6cf3ab11949f1a66d41e: Status 404 returned error can't find the container with id 4a739e459ed354cfdca54cd4d0d96c1c7e1e4a8a61ce6cf3ab11949f1a66d41e
Oct 30 02:48:56.017527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:56.017483    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" event={"ID":"6627dcdc-059c-41ba-b173-f91031ad4db1","Type":"ContainerStarted","Data":"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"}
Oct 30 02:48:56.018050 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:56.017538    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" event={"ID":"6627dcdc-059c-41ba-b173-f91031ad4db1","Type":"ContainerStarted","Data":"4a739e459ed354cfdca54cd4d0d96c1c7e1e4a8a61ce6cf3ab11949f1a66d41e"}
Oct 30 02:48:56.018050 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:56.017584    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:48:56.088146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:56.088096    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" podStartSLOduration=2.088081972 podStartE2EDuration="2.088081972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:56.086485468 +0000 UTC m=+475.160656261" watchObservedRunningTime="2026-10-30 02:48:56.088081972 +0000 UTC m=+475.162252765"
Oct 30 02:48:57.064633 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:48:57.064603    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce30dee-4ca7-4fbf-992a-27538b10f933" path="/var/lib/kubelet/pods/bce30dee-4ca7-4fbf-992a-27538b10f933/volumes"
Oct 30 02:49:07.024757 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.024724    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 02:49:07.112704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.112665    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:49:07.113052 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.112993    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" podUID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" containerName="manager" containerID="cri-o://d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0" gracePeriod=10
Oct 30 02:49:07.374286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.374258    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:49:07.479325 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.479296    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\" (UniqueName: \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\") pod \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") "
Oct 30 02:49:07.479496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.479366    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\") pod \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\" (UID: \"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4\") "
Oct 30 02:49:07.479767 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.479737    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume" pod "36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" (UID: "36a7f0a9-9a2d-4af7-b6d0-51e5053700e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:49:07.481437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.481411    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd" pod "36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" (UID: "36a7f0a9-9a2d-4af7-b6d0-51e5053700e4"). InnerVolumeSpecName "kube-api-access-n5cdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:49:07.580409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.580383    4214 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-extensions-socket-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:49:07.580409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:07.580407    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cdd\" (UniqueName: \"kubernetes.io/projected/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4-kube-api-access-n5cdd\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:49:08.073549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.073513    4214 generic.go:356] "Generic (PLEG): container finished" podID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" containerID="d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0" exitCode=0
Oct 30 02:49:08.073981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.073554    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"
Oct 30 02:49:08.073981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.073595    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" event={"ID":"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4","Type":"ContainerDied","Data":"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"}
Oct 30 02:49:08.073981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.073649    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m" event={"ID":"36a7f0a9-9a2d-4af7-b6d0-51e5053700e4","Type":"ContainerDied","Data":"318596c97b9d5e210876bd4dbeef00bbea7c36c8982820ecc3326d4f6459adbd"}
Oct 30 02:49:08.073981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.073678    4214 scope.go:122] "RemoveContainer" containerID="d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"
Oct 30 02:49:08.082597 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.082581    4214 scope.go:122] "RemoveContainer" containerID="d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"
Oct 30 02:49:08.082865 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:49:08.082848    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0\": container with ID starting with d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0 not found: ID does not exist" containerID="d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"
Oct 30 02:49:08.082908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.082877    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0"} err="failed to get container status \"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0\": rpc error: code = NotFound desc = could not find container \"d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0\": container with ID starting with d03a6f9fc9e67d4697b6d5438c662ff5c6f2e9f2517ca04ad93ee183e70af9a0 not found: ID does not exist"
Oct 30 02:49:08.144513 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.144486    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:49:08.157166 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:08.157135    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-b9n6m"]
Oct 30 02:49:09.063908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:09.063867    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" path="/var/lib/kubelet/pods/36a7f0a9-9a2d-4af7-b6d0-51e5053700e4/volumes"
Oct 30 02:49:09.162619 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:49:09.162583    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:49:24.960219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.960143    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:24.960616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.960504    4214 cpu_manager.go:414] "containerMap: removing container" podUID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" containerName="manager"
Oct 30 02:49:24.960616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.960515    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" containerName="manager"
Oct 30 02:49:24.960616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.960574    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a7f0a9-9a2d-4af7-b6d0-51e5053700e4" containerName="manager"
Oct 30 02:49:24.963327 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.963310    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:24.982095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.982070    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 02:49:24.984496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.984471    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:24.995089 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:24.995068    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-88pp5\""
Oct 30 02:49:25.026023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.025998    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfz6b\" (UniqueName: \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.026171 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.026051    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.090603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.090571    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:25.126682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.126648    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfz6b\" (UniqueName: \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.126860 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.126706    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.127571 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.127551    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.151665 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.151637    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfz6b\" (UniqueName: \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\") pod \"limitador-limitador-74dfcfb894-8pktt\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.276012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.275913    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:25.574367 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:25.574307    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:25.576664 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:49:25.576637    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945ebf5a_a59c_4c36_948f_1d08ee84e0b1.slice/crio-39663f33f2da35ab424d44f7670854e9000a3081d52423eaa283d5b3fc1f6ffc WatchSource:0}: Error finding container 39663f33f2da35ab424d44f7670854e9000a3081d52423eaa283d5b3fc1f6ffc: Status 404 returned error can't find the container with id 39663f33f2da35ab424d44f7670854e9000a3081d52423eaa283d5b3fc1f6ffc
Oct 30 02:49:26.160059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:26.160019    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" event={"ID":"945ebf5a-a59c-4c36-948f-1d08ee84e0b1","Type":"ContainerStarted","Data":"39663f33f2da35ab424d44f7670854e9000a3081d52423eaa283d5b3fc1f6ffc"}
Oct 30 02:49:27.166290 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:27.166255    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" event={"ID":"945ebf5a-a59c-4c36-948f-1d08ee84e0b1","Type":"ContainerStarted","Data":"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"}
Oct 30 02:49:27.166713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:27.166342    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:27.209246 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:27.209177    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" podStartSLOduration=2.347788439 podStartE2EDuration="3.209163042s" totalImagesPullingTime="861.374603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:27.207647443 +0000 UTC m=+506.281818235" watchObservedRunningTime="2026-10-30 02:49:27.209163042 +0000 UTC m=+506.283333834"
Oct 30 02:49:38.172190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:38.172158    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:39.163134 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:49:39.163098    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:49:39.515722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:39.515646    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:39.516172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:39.515815    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" podUID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" containerName="limitador" containerID="cri-o://911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9" gracePeriod=30
Oct 30 02:49:40.057791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.057766    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:40.146499 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.146413    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\" (UniqueName: \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\") pod \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") "
Oct 30 02:49:40.146499 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.146467    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\" (UniqueName: \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\") pod \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\" (UID: \"945ebf5a-a59c-4c36-948f-1d08ee84e0b1\") "
Oct 30 02:49:40.146820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.146795    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file" pod "945ebf5a-a59c-4c36-948f-1d08ee84e0b1" (UID: "945ebf5a-a59c-4c36-948f-1d08ee84e0b1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:49:40.148519 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.148497    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b" pod "945ebf5a-a59c-4c36-948f-1d08ee84e0b1" (UID: "945ebf5a-a59c-4c36-948f-1d08ee84e0b1"). InnerVolumeSpecName "kube-api-access-mfz6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:49:40.226266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.226224    4214 generic.go:356] "Generic (PLEG): container finished" podID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" containerID="911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9" exitCode=0
Oct 30 02:49:40.226413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.226280    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt"
Oct 30 02:49:40.226413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.226316    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" event={"ID":"945ebf5a-a59c-4c36-948f-1d08ee84e0b1","Type":"ContainerDied","Data":"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"}
Oct 30 02:49:40.226413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.226366    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8pktt" event={"ID":"945ebf5a-a59c-4c36-948f-1d08ee84e0b1","Type":"ContainerDied","Data":"39663f33f2da35ab424d44f7670854e9000a3081d52423eaa283d5b3fc1f6ffc"}
Oct 30 02:49:40.226413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.226388    4214 scope.go:122] "RemoveContainer" containerID="911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"
Oct 30 02:49:40.235001 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.234979    4214 scope.go:122] "RemoveContainer" containerID="911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"
Oct 30 02:49:40.235279 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:49:40.235258    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9\": container with ID starting with 911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9 not found: ID does not exist" containerID="911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"
Oct 30 02:49:40.235366 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.235287    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9"} err="failed to get container status \"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9\": rpc error: code = NotFound desc = could not find container \"911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9\": container with ID starting with 911b54a3c6a68dcf5e200125b3aba17acca2f976011fd6c3bb5dbf539df0e8c9 not found: ID does not exist"
Oct 30 02:49:40.247859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.247837    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfz6b\" (UniqueName: \"kubernetes.io/projected/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-kube-api-access-mfz6b\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:49:40.247859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.247859    4214 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/945ebf5a-a59c-4c36-948f-1d08ee84e0b1-config-file\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:49:40.283437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.283406    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:40.293060 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:40.293033    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8pktt"]
Oct 30 02:49:41.063759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:41.063727    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" path="/var/lib/kubelet/pods/945ebf5a-a59c-4c36-948f-1d08ee84e0b1/volumes"
Oct 30 02:49:46.758944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.758905    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"]
Oct 30 02:49:46.759462 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.759404    4214 cpu_manager.go:414] "containerMap: removing container" podUID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" containerName="limitador"
Oct 30 02:49:46.759462 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.759423    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" containerName="limitador"
Oct 30 02:49:46.759582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.759525    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="945ebf5a-a59c-4c36-948f-1d08ee84e0b1" containerName="limitador"
Oct 30 02:49:46.762572 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.762550    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:46.779791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.779767    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"]
Oct 30 02:49:46.790979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.790955    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 02:49:46.790979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.790973    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9pzjs\""
Oct 30 02:49:46.791142 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.791007    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 02:49:46.791142 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.791041    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 02:49:46.905759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.905719    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db49e973-a835-45e7-8513-9f5315159095-data\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:46.905909 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:46.905767    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq6s\" (UniqueName: \"kubernetes.io/projected/db49e973-a835-45e7-8513-9f5315159095-kube-api-access-vpq6s\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.006645 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.006611    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db49e973-a835-45e7-8513-9f5315159095-data\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.006801 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.006651    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq6s\" (UniqueName: \"kubernetes.io/projected/db49e973-a835-45e7-8513-9f5315159095-kube-api-access-vpq6s\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.006998 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.006972    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db49e973-a835-45e7-8513-9f5315159095-data\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.027453 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.027395    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq6s\" (UniqueName: \"kubernetes.io/projected/db49e973-a835-45e7-8513-9f5315159095-kube-api-access-vpq6s\") pod \"postgres-7cdbf6dc44-2lj4q\" (UID: \"db49e973-a835-45e7-8513-9f5315159095\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.075350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.075321    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:47.370308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:47.370190    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"]
Oct 30 02:49:47.372724 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:49:47.372697    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb49e973_a835_45e7_8513_9f5315159095.slice/crio-1d301a63c6282b936c15f1033fd918ad663bbb54532c0b18b85a5a2ef320a7f8 WatchSource:0}: Error finding container 1d301a63c6282b936c15f1033fd918ad663bbb54532c0b18b85a5a2ef320a7f8: Status 404 returned error can't find the container with id 1d301a63c6282b936c15f1033fd918ad663bbb54532c0b18b85a5a2ef320a7f8
Oct 30 02:49:48.269400 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:48.269359    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q" event={"ID":"db49e973-a835-45e7-8513-9f5315159095","Type":"ContainerStarted","Data":"1d301a63c6282b936c15f1033fd918ad663bbb54532c0b18b85a5a2ef320a7f8"}
Oct 30 02:49:52.291011 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:52.290919    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q" event={"ID":"db49e973-a835-45e7-8513-9f5315159095","Type":"ContainerStarted","Data":"262096f0d3deec5fa681d289f880351a86b576d3134087ff730df7aac6df831d"}
Oct 30 02:49:52.291011 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:52.290977    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:49:52.333061 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:52.333009    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q" podStartSLOduration=1.8506581949999998 podStartE2EDuration="6.332996785s" totalImagesPullingTime="4.48233859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:52.331663002 +0000 UTC m=+531.405833792" watchObservedRunningTime="2026-10-30 02:49:52.332996785 +0000 UTC m=+531.407167577"
Oct 30 02:49:58.325370 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:49:58.325342    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2lj4q"
Oct 30 02:50:08.433185 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.433151    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:08.438664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.438648    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.468718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.468693    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 02:50:08.468718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.468693    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 02:50:08.468918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.468817    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7hglq\""
Oct 30 02:50:08.468918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.468872    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 02:50:08.471963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.471939    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:08.584203 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.584169    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.584374 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.584247    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnz9\" (UniqueName: \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.584374 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.584328    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.685117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.685034    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnz9\" (UniqueName: \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.685117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.685081    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.685384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.685153    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.687647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.687620    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.687647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.687637    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.704172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.704150    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnz9\" (UniqueName: \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\") pod \"maas-controller-577f596567-x685j\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") " pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.750943 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.750911    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:08.796238 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.796192    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 02:50:08.801804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.801778    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:08.820792 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.820511    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 02:50:08.988622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.988535    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:08.988622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.988582    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:08.988811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:08.988660    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779dg\" (UniqueName: \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.053129 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.053074    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:09.055811 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:50:09.055777    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2a52b_b3bb_4d21_abcf_16743e149d6c.slice/crio-859c8f45f05b0031017ebf5f3872269fe6119543e58e53489b1b19c0b17b40ef WatchSource:0}: Error finding container 859c8f45f05b0031017ebf5f3872269fe6119543e58e53489b1b19c0b17b40ef: Status 404 returned error can't find the container with id 859c8f45f05b0031017ebf5f3872269fe6119543e58e53489b1b19c0b17b40ef
Oct 30 02:50:09.090047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.090018    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779dg\" (UniqueName: \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.090172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.090095    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.090172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.090114    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.092473 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.092448    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.092567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.092478    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.109845 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.109824    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779dg\" (UniqueName: \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\") pod \"maas-controller-6658f68bfc-xts57\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") " pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.125683 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.125666    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:09.164261 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:50:09.164210    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:50:09.376063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.376021    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-x685j" event={"ID":"40d2a52b-b3bb-4d21-abcf-16743e149d6c","Type":"ContainerStarted","Data":"859c8f45f05b0031017ebf5f3872269fe6119543e58e53489b1b19c0b17b40ef"}
Oct 30 02:50:09.425494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:09.425469    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 02:50:09.427636 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:50:09.427606    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1b3588_f564_430d_b012_411cd460d36d.slice/crio-4d77cecf41308188eb86350356de1e82bb9b85abf2eedd859df78b43eacbdaad WatchSource:0}: Error finding container 4d77cecf41308188eb86350356de1e82bb9b85abf2eedd859df78b43eacbdaad: Status 404 returned error can't find the container with id 4d77cecf41308188eb86350356de1e82bb9b85abf2eedd859df78b43eacbdaad
Oct 30 02:50:10.389096 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:10.389052    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-xts57" event={"ID":"8f1b3588-f564-430d-b012-411cd460d36d","Type":"ContainerStarted","Data":"4d77cecf41308188eb86350356de1e82bb9b85abf2eedd859df78b43eacbdaad"}
Oct 30 02:50:12.401312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.401275    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-x685j" event={"ID":"40d2a52b-b3bb-4d21-abcf-16743e149d6c","Type":"ContainerStarted","Data":"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"}
Oct 30 02:50:12.401750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.401385    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:12.402928 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.402905    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-xts57" event={"ID":"8f1b3588-f564-430d-b012-411cd460d36d","Type":"ContainerStarted","Data":"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"}
Oct 30 02:50:12.403037 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.402955    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:12.442895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.442840    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-x685j" podStartSLOduration=1.508129584 podStartE2EDuration="4.442824345s" totalImagesPullingTime="2.934694761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:12.442189693 +0000 UTC m=+551.516360485" watchObservedRunningTime="2026-10-30 02:50:12.442824345 +0000 UTC m=+551.516995141"
Oct 30 02:50:12.485577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:12.485491    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6658f68bfc-xts57" podStartSLOduration=1.8103311500000001 podStartE2EDuration="4.485450392s" totalImagesPullingTime="2.675119242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:12.485158579 +0000 UTC m=+551.559329363" watchObservedRunningTime="2026-10-30 02:50:12.485450392 +0000 UTC m=+551.559621185"
Oct 30 02:50:15.662868 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.662834    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"]
Oct 30 02:50:15.666482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.666467    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.694947 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.694914    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 02:50:15.695254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.694931    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 02:50:15.695254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.694940    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-r2xp4\""
Oct 30 02:50:15.695396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.695044    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 02:50:15.697052 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.697034    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"]
Oct 30 02:50:15.748195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.748159    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.748195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.748196    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.748425 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.748257    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.748425 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.748294    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwx98\" (UniqueName: \"kubernetes.io/projected/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-kube-api-access-cwx98\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.849221 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.849178    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.849221 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.849215    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwx98\" (UniqueName: \"kubernetes.io/projected/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-kube-api-access-cwx98\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.849494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.849311    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.849494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.849330    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.850077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.850047    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.850197 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.850176    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.851567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.851544    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.869108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.869085    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwx98\" (UniqueName: \"kubernetes.io/projected/c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44-kube-api-access-cwx98\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-gtpxx\" (UID: \"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:15.983524 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:15.983436    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:16.279563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:16.279520    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"]
Oct 30 02:50:16.281716 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:50:16.281685    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cb0eed_1e68_4dbf_b45b_6de4c48d9f44.slice/crio-aa1bb2d6e9d42ebb401c7e33cb944389a3e8f33a4f59961220becedeb7008297 WatchSource:0}: Error finding container aa1bb2d6e9d42ebb401c7e33cb944389a3e8f33a4f59961220becedeb7008297: Status 404 returned error can't find the container with id aa1bb2d6e9d42ebb401c7e33cb944389a3e8f33a4f59961220becedeb7008297
Oct 30 02:50:16.422406 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:16.422372    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx" event={"ID":"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44","Type":"ContainerStarted","Data":"aa1bb2d6e9d42ebb401c7e33cb944389a3e8f33a4f59961220becedeb7008297"}
Oct 30 02:50:23.410635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.410601    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:23.599857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.410691    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 02:50:23.599857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.521578    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:23.599857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.521783    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-x685j" podUID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" containerName="manager" containerID="cri-o://fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f" gracePeriod=30
Oct 30 02:50:23.827586 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.827549    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:23.924805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.924771    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\" (UniqueName: \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\") pod \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") "
Oct 30 02:50:23.924971 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.924856    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\") pod \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") "
Oct 30 02:50:23.924971 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.924886    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\") pod \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\" (UID: \"40d2a52b-b3bb-4d21-abcf-16743e149d6c\") "
Oct 30 02:50:23.927360 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.927330    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs" pod "40d2a52b-b3bb-4d21-abcf-16743e149d6c" (UID: "40d2a52b-b3bb-4d21-abcf-16743e149d6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:50:23.927508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.927446    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert" pod "40d2a52b-b3bb-4d21-abcf-16743e149d6c" (UID: "40d2a52b-b3bb-4d21-abcf-16743e149d6c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:50:23.927508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:23.927478    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9" pod "40d2a52b-b3bb-4d21-abcf-16743e149d6c" (UID: "40d2a52b-b3bb-4d21-abcf-16743e149d6c"). InnerVolumeSpecName "kube-api-access-bbnz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:50:24.026343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.026262    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:50:24.026343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.026302    4214 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d2a52b-b3bb-4d21-abcf-16743e149d6c-webhook-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:50:24.026343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.026318    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnz9\" (UniqueName: \"kubernetes.io/projected/40d2a52b-b3bb-4d21-abcf-16743e149d6c-kube-api-access-bbnz9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:50:24.479058 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.479019    4214 generic.go:356] "Generic (PLEG): container finished" podID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" containerID="fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f" exitCode=0
Oct 30 02:50:24.479522 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.479068    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-x685j"
Oct 30 02:50:24.479522 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.479097    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-x685j" event={"ID":"40d2a52b-b3bb-4d21-abcf-16743e149d6c","Type":"ContainerDied","Data":"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"}
Oct 30 02:50:24.479522 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.479154    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-x685j" event={"ID":"40d2a52b-b3bb-4d21-abcf-16743e149d6c","Type":"ContainerDied","Data":"859c8f45f05b0031017ebf5f3872269fe6119543e58e53489b1b19c0b17b40ef"}
Oct 30 02:50:24.479522 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.479175    4214 scope.go:122] "RemoveContainer" containerID="fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"
Oct 30 02:50:24.491435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.491341    4214 scope.go:122] "RemoveContainer" containerID="fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"
Oct 30 02:50:24.493526 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:50:24.493500    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f\": container with ID starting with fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f not found: ID does not exist" containerID="fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"
Oct 30 02:50:24.493625 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.493531    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f"} err="failed to get container status \"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f\": rpc error: code = NotFound desc = could not find container \"fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f\": container with ID starting with fa4135a81852ff7a58b346e0d222651bde139af51e5cc063d88d7a265b0ba37f not found: ID does not exist"
Oct 30 02:50:24.533175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.533148    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:24.546016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:24.545987    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-x685j"]
Oct 30 02:50:25.064651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:25.064620    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" path="/var/lib/kubelet/pods/40d2a52b-b3bb-4d21-abcf-16743e149d6c/volumes"
Oct 30 02:50:27.498363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:27.498329    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx" event={"ID":"c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44","Type":"ContainerStarted","Data":"ae3d8936b67a69aa73e68121f59bd110f621577bf37bcca1a7466ede7d45d616"}
Oct 30 02:50:27.498803 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:27.498463    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:27.541802 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:27.541752    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx" podStartSLOduration=2.389992229 podStartE2EDuration="12.541738967s" totalImagesPullingTime="10.151746738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:27.539915072 +0000 UTC m=+566.614085868" watchObservedRunningTime="2026-10-30 02:50:27.541738967 +0000 UTC m=+566.615909788"
Oct 30 02:50:38.509122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:50:38.509091    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-gtpxx"
Oct 30 02:50:39.162829 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:50:39.162793    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:51:09.162946 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:51:09.162907    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:51:39.162872 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:51:39.162835    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:51:49.329549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.329512    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"]
Oct 30 02:51:49.330035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.330016    4214 cpu_manager.go:414] "containerMap: removing container" podUID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" containerName="manager"
Oct 30 02:51:49.330085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.330040    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" containerName="manager"
Oct 30 02:51:49.330176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.330164    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d2a52b-b3bb-4d21-abcf-16743e149d6c" containerName="manager"
Oct 30 02:51:49.332823 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.332803    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.349166 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.349143    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9xfvj\""
Oct 30 02:51:49.362273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.362251    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 02:51:49.362273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.362270    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 02:51:49.364103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.364085    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"]
Oct 30 02:51:49.438277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438247    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv29x\" (UniqueName: \"kubernetes.io/projected/2af9b4df-18a9-404e-b4f2-e1df80947697-kube-api-access-qv29x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.438416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438285    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af9b4df-18a9-404e-b4f2-e1df80947697-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.438416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438376    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.438416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438399    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.438536 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438493    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.438592 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.438576    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.449157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.449135    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 02:51:49.538971 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.538941    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539016    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539059    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv29x\" (UniqueName: \"kubernetes.io/projected/2af9b4df-18a9-404e-b4f2-e1df80947697-kube-api-access-qv29x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539078    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af9b4df-18a9-404e-b4f2-e1df80947697-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539104    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539145 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539127    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539458 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539528 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539499    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.539579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.539535    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.541278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.541256    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af9b4df-18a9-404e-b4f2-e1df80947697-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.541728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.541707    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af9b4df-18a9-404e-b4f2-e1df80947697-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.558916 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.558888    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv29x\" (UniqueName: \"kubernetes.io/projected/2af9b4df-18a9-404e-b4f2-e1df80947697-kube-api-access-qv29x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-cc52z\" (UID: \"2af9b4df-18a9-404e-b4f2-e1df80947697\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.645076 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.645008    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:49.943193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.943166    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"]
Oct 30 02:51:49.944971 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:51:49.944949    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af9b4df_18a9_404e_b4f2_e1df80947697.slice/crio-08f71900d816d0a67aee95ebcc3b3b488fc4db15ecc06eca392fc24acae4f9c5 WatchSource:0}: Error finding container 08f71900d816d0a67aee95ebcc3b3b488fc4db15ecc06eca392fc24acae4f9c5: Status 404 returned error can't find the container with id 08f71900d816d0a67aee95ebcc3b3b488fc4db15ecc06eca392fc24acae4f9c5
Oct 30 02:51:49.946125 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:49.946110    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:51:50.898442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:50.898399    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z" event={"ID":"2af9b4df-18a9-404e-b4f2-e1df80947697","Type":"ContainerStarted","Data":"08f71900d816d0a67aee95ebcc3b3b488fc4db15ecc06eca392fc24acae4f9c5"}
Oct 30 02:51:51.904755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:51.904720    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z" event={"ID":"2af9b4df-18a9-404e-b4f2-e1df80947697","Type":"ContainerStarted","Data":"90694cc804671d1736bae7fc241de5eee919992d919cf060b953bca5f4a38d96"}
Oct 30 02:51:51.953727 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:51.953635    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z" podStartSLOduration=1.894203329 podStartE2EDuration="2.953613889s" totalImagesPullingTime="1.05941056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:51:51.95315052 +0000 UTC m=+651.027321314" watchObservedRunningTime="2026-10-30 02:51:51.953613889 +0000 UTC m=+651.027784725"
Oct 30 02:51:52.645959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:52.645915    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:55.166056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.166015    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"]
Oct 30 02:51:55.174133 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.174108    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.188963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.188940    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"]
Oct 30 02:51:55.199475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.199452    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 02:51:55.289068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289027    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.289068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289063    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzz7\" (UniqueName: \"kubernetes.io/projected/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-kube-api-access-dmzz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.289342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289088    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.289342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289210    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.289342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289245    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.289342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.289262    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390458    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390499    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzz7\" (UniqueName: \"kubernetes.io/projected/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-kube-api-access-dmzz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390531    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390567    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390597    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390619    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.390959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390934    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.391027 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.390969    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.391067 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.391036    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.392771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.392753    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.393044 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.393025    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.410147 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.410114    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzz7\" (UniqueName: \"kubernetes.io/projected/63d54bdc-70b6-4c6c-8ab2-4dcab5038bab-kube-api-access-dmzz7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4\" (UID: \"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.489210 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.489140    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:55.787885 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.787861    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"]
Oct 30 02:51:55.789582 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:51:55.789556    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d54bdc_70b6_4c6c_8ab2_4dcab5038bab.slice/crio-d0c6594f6bf364f72b95d60fcbebbf824378782dd2feb8fb671af91829410164 WatchSource:0}: Error finding container d0c6594f6bf364f72b95d60fcbebbf824378782dd2feb8fb671af91829410164: Status 404 returned error can't find the container with id d0c6594f6bf364f72b95d60fcbebbf824378782dd2feb8fb671af91829410164
Oct 30 02:51:55.923666 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:55.923629    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4" event={"ID":"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab","Type":"ContainerStarted","Data":"d0c6594f6bf364f72b95d60fcbebbf824378782dd2feb8fb671af91829410164"}
Oct 30 02:51:56.929952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:56.929917    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4" event={"ID":"63d54bdc-70b6-4c6c-8ab2-4dcab5038bab","Type":"ContainerStarted","Data":"a9995900febdfb17a3f4f75e429e3044793ee42ab583d3ae52fd4d4071fbcd93"}
Oct 30 02:51:56.971698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:56.971646    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4" podStartSLOduration=1.785879278 podStartE2EDuration="1.971634729s" totalImagesPullingTime="185.755451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:51:56.969781417 +0000 UTC m=+656.043952212" watchObservedRunningTime="2026-10-30 02:51:56.971634729 +0000 UTC m=+656.045805565"
Oct 30 02:51:57.490053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:57.490016    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:51:59.645381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:59.645341    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:59.657752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:59.657724    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:51:59.955266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:51:59.955181    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-cc52z"
Oct 30 02:52:00.479921 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.479885    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"]
Oct 30 02:52:00.483033 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.483003    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.500110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.500089    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 02:52:00.515160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.515134    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"]
Oct 30 02:52:00.640673 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640637    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.640831 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640765    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.640873 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640829    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44nx\" (UniqueName: \"kubernetes.io/projected/74cb6b35-cc28-4b86-be20-3d994dd660bc-kube-api-access-l44nx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.640912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640890    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.640943 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640913    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cb6b35-cc28-4b86-be20-3d994dd660bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.640989 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.640946    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.742482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742406    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.742482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742455    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.742482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742482    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44nx\" (UniqueName: \"kubernetes.io/projected/74cb6b35-cc28-4b86-be20-3d994dd660bc-kube-api-access-l44nx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742551    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742887    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742921    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.742577    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cb6b35-cc28-4b86-be20-3d994dd660bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743458 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.743348    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.743679 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.743650    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.745004 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.744965    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cb6b35-cc28-4b86-be20-3d994dd660bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.745953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.745932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cb6b35-cc28-4b86-be20-3d994dd660bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.764039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.764009    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44nx\" (UniqueName: \"kubernetes.io/projected/74cb6b35-cc28-4b86-be20-3d994dd660bc-kube-api-access-l44nx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt\" (UID: \"74cb6b35-cc28-4b86-be20-3d994dd660bc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:00.796141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:00.796113    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:01.097111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:01.097088    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"]
Oct 30 02:52:01.098675 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:01.098650    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cb6b35_cc28_4b86_be20_3d994dd660bc.slice/crio-a036d1f77d2ac5738eb32cb1a4dedbd96132acb91f6bb4fd6e80f9c2a6d47b9f WatchSource:0}: Error finding container a036d1f77d2ac5738eb32cb1a4dedbd96132acb91f6bb4fd6e80f9c2a6d47b9f: Status 404 returned error can't find the container with id a036d1f77d2ac5738eb32cb1a4dedbd96132acb91f6bb4fd6e80f9c2a6d47b9f
Oct 30 02:52:01.955560 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:01.955520    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt" event={"ID":"74cb6b35-cc28-4b86-be20-3d994dd660bc","Type":"ContainerStarted","Data":"841e41f6faa526a733b57f4159fa71578e41cb97c538f650b4f700bf87f1f8eb"}
Oct 30 02:52:01.955933 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:01.955574    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt" event={"ID":"74cb6b35-cc28-4b86-be20-3d994dd660bc","Type":"ContainerStarted","Data":"a036d1f77d2ac5738eb32cb1a4dedbd96132acb91f6bb4fd6e80f9c2a6d47b9f"}
Oct 30 02:52:01.997603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:01.997557    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt" podStartSLOduration=1.825619393 podStartE2EDuration="1.997543504s" totalImagesPullingTime="171.924111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:01.995656702 +0000 UTC m=+661.069827496" watchObservedRunningTime="2026-10-30 02:52:01.997543504 +0000 UTC m=+661.071714296"
Oct 30 02:52:02.639063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.639027    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"]
Oct 30 02:52:02.642091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.642068    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659159 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659129    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmqv\" (UniqueName: \"kubernetes.io/projected/04056afb-ee76-4d83-a900-95fc26f11e7c-kube-api-access-slmqv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659173    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659215    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659339    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04056afb-ee76-4d83-a900-95fc26f11e7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659365    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.659447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.659417    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.672986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.672962    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 02:52:02.676563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.676533    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"]
Oct 30 02:52:02.760527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.760489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.760934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.760902    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmqv\" (UniqueName: \"kubernetes.io/projected/04056afb-ee76-4d83-a900-95fc26f11e7c-kube-api-access-slmqv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.761087 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.760980    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.761087 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.761037    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.761087 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.761082    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04056afb-ee76-4d83-a900-95fc26f11e7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.761270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.761113    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.762384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.761759    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.762384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.761842    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.762384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.762112    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.768986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.768434    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04056afb-ee76-4d83-a900-95fc26f11e7c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.774023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.773987    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04056afb-ee76-4d83-a900-95fc26f11e7c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.781268 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.781219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmqv\" (UniqueName: \"kubernetes.io/projected/04056afb-ee76-4d83-a900-95fc26f11e7c-kube-api-access-slmqv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fmtkt\" (UID: \"04056afb-ee76-4d83-a900-95fc26f11e7c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:02.796758 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.796736    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:02.958953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:02.958861    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:03.256608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:03.256584    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"]
Oct 30 02:52:03.258768 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:03.258737    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04056afb_ee76_4d83_a900_95fc26f11e7c.slice/crio-45f91dce599e415a7bbf608f39381ba13179ba8a77266333326eaf1df78cd3ad WatchSource:0}: Error finding container 45f91dce599e415a7bbf608f39381ba13179ba8a77266333326eaf1df78cd3ad: Status 404 returned error can't find the container with id 45f91dce599e415a7bbf608f39381ba13179ba8a77266333326eaf1df78cd3ad
Oct 30 02:52:03.967018 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:03.966970    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt" event={"ID":"04056afb-ee76-4d83-a900-95fc26f11e7c","Type":"ContainerStarted","Data":"4ea53864a81a13f3e590238df6d8a3fa3988e40f68b15f4e044b3d6f7ac5c517"}
Oct 30 02:52:03.967406 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:03.967041    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt" event={"ID":"04056afb-ee76-4d83-a900-95fc26f11e7c","Type":"ContainerStarted","Data":"45f91dce599e415a7bbf608f39381ba13179ba8a77266333326eaf1df78cd3ad"}
Oct 30 02:52:04.009041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:04.008993    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt" podStartSLOduration=1.8467502489999998 podStartE2EDuration="2.008979341s" totalImagesPullingTime="162.229092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:04.007457717 +0000 UTC m=+663.081628511" watchObservedRunningTime="2026-10-30 02:52:04.008979341 +0000 UTC m=+663.083150133"
Oct 30 02:52:04.959981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:04.959948    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:05.489300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:05.489261    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:52:05.502213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:05.502191    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:52:05.987605 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:05.987574    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4"
Oct 30 02:52:09.163486 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:52:09.163454    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:52:10.797059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:10.797020    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:10.809503 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:10.809481    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:11.013357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:11.013327    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt"
Oct 30 02:52:12.543412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.543380    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:12.546890 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.546871    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.576143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.576116    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m8lgr\""
Oct 30 02:52:12.576143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.576140    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 02:52:12.576342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.576147    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 02:52:12.578289 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.578270    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:12.650252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.650208    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.650412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.650349    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96sdr\" (UniqueName: \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.751162 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.751126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96sdr\" (UniqueName: \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.751354 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.751198    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.753584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.753557    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.771570 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.771538    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96sdr\" (UniqueName: \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\") pod \"ai-gateway-controller-55b4c89cbd-7np9b\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.859981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.859902    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:12.959270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.959239    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:12.971754 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:12.971731    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:13.024370 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:13.024345    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt"
Oct 30 02:52:13.153647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:13.153593    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:13.156145 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:13.156117    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe482e9_b27d_4cf6_a01f_5601aef5409f.slice/crio-5edbf3ab4952d882e73151b0d92ece745a2ef4a3b3be4d5f2ff9796d8b05b2b4 WatchSource:0}: Error finding container 5edbf3ab4952d882e73151b0d92ece745a2ef4a3b3be4d5f2ff9796d8b05b2b4: Status 404 returned error can't find the container with id 5edbf3ab4952d882e73151b0d92ece745a2ef4a3b3be4d5f2ff9796d8b05b2b4
Oct 30 02:52:14.026862 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:14.026781    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" event={"ID":"9fe482e9-b27d-4cf6-a01f-5601aef5409f","Type":"ContainerStarted","Data":"5edbf3ab4952d882e73151b0d92ece745a2ef4a3b3be4d5f2ff9796d8b05b2b4"}
Oct 30 02:52:16.041838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:16.041731    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" event={"ID":"9fe482e9-b27d-4cf6-a01f-5601aef5409f","Type":"ContainerStarted","Data":"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"}
Oct 30 02:52:16.041838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:16.041799    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:16.082482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:16.082432    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" podStartSLOduration=1.61362448 podStartE2EDuration="4.082418429s" totalImagesPullingTime="2.468793949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:16.081299598 +0000 UTC m=+675.155470390" watchObservedRunningTime="2026-10-30 02:52:16.082418429 +0000 UTC m=+675.156589225"
Oct 30 02:52:17.038693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.038656    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"]
Oct 30 02:52:17.042297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.042270    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.061103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.061080    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hch59\""
Oct 30 02:52:17.089540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.089517    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 02:52:17.093668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.093648    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.093766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.093679    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq67\" (UniqueName: \"kubernetes.io/projected/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-kube-api-access-fvq67\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.108551 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.108520    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"]
Oct 30 02:52:17.195185 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.195140    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.195464 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.195433    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq67\" (UniqueName: \"kubernetes.io/projected/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-kube-api-access-fvq67\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.196265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.196219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.222614 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.222589    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq67\" (UniqueName: \"kubernetes.io/projected/de64f1d2-dc08-4bc0-9c69-f7b588b88c46-kube-api-access-fvq67\") pod \"payload-pre-processing-7b98bb678c-8tfqx\" (UID: \"de64f1d2-dc08-4bc0-9c69-f7b588b88c46\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.358956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.358923    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:17.663334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:17.663302    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"]
Oct 30 02:52:17.665320 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:17.665290    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde64f1d2_dc08_4bc0_9c69_f7b588b88c46.slice/crio-68ff8cc8d78431dfccd8a9d7d1b1882b7f05ad750ea3f628613a48c9efcc5918 WatchSource:0}: Error finding container 68ff8cc8d78431dfccd8a9d7d1b1882b7f05ad750ea3f628613a48c9efcc5918: Status 404 returned error can't find the container with id 68ff8cc8d78431dfccd8a9d7d1b1882b7f05ad750ea3f628613a48c9efcc5918
Oct 30 02:52:18.053375 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:18.053335    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx" event={"ID":"de64f1d2-dc08-4bc0-9c69-f7b588b88c46","Type":"ContainerStarted","Data":"68ff8cc8d78431dfccd8a9d7d1b1882b7f05ad750ea3f628613a48c9efcc5918"}
Oct 30 02:52:20.066019 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:20.065981    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx" event={"ID":"de64f1d2-dc08-4bc0-9c69-f7b588b88c46","Type":"ContainerStarted","Data":"46f05c63a24b07e4e0ad2ce602bf525241fc6955edc57273c7ff5a1ec3538791"}
Oct 30 02:52:20.066450 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:20.066181    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:20.108788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:20.108742    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx" podStartSLOduration=2.161473489 podStartE2EDuration="4.108724306s" totalImagesPullingTime="1.947250817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:20.107040509 +0000 UTC m=+679.181211304" watchObservedRunningTime="2026-10-30 02:52:20.108724306 +0000 UTC m=+679.182895099"
Oct 30 02:52:26.975919 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:26.975881    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:52:26.982069 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:26.982046    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.019615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.019589    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:52:27.048651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.048629    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:27.075175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.075145    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wg9w9\""
Oct 30 02:52:27.089631 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089606    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.089737 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089652    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.089737 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089696    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.089843 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089825    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdxf\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.089888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089873    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.089943 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089929    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.090004 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.089993    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.090040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.090011    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.090040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.090028    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191319 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191291    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191319 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191321    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191373    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191400    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191424    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191461    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdxf\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191510    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191879 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191546    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191879 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191734    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.191879 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191849    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.192047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.191992    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.192128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.192108    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.192196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.192170    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.193773 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.193752    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.193862 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.193811    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.214198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.214175    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.214472 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.214450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdxf\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-v5g8v\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.296352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.296331    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:27.365260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.365213    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74685b6749-zmk6j"]
Oct 30 02:52:27.370729 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.370709    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.482936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.482905    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74685b6749-zmk6j"]
Oct 30 02:52:27.493882 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.493857    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088fc29-fd63-4aa4-b655-c8482f25fbdb-metrics-certs\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.494006 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.493904    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgtt\" (UniqueName: \"kubernetes.io/projected/0088fc29-fd63-4aa4-b655-c8482f25fbdb-kube-api-access-gpgtt\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.594404 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.594340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088fc29-fd63-4aa4-b655-c8482f25fbdb-metrics-certs\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.594404 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.594382    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgtt\" (UniqueName: \"kubernetes.io/projected/0088fc29-fd63-4aa4-b655-c8482f25fbdb-kube-api-access-gpgtt\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.596649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.596627    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088fc29-fd63-4aa4-b655-c8482f25fbdb-metrics-certs\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.618154 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.618125    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgtt\" (UniqueName: \"kubernetes.io/projected/0088fc29-fd63-4aa4-b655-c8482f25fbdb-kube-api-access-gpgtt\") pod \"ai-gateway-controller-74685b6749-zmk6j\" (UID: \"0088fc29-fd63-4aa4-b655-c8482f25fbdb\") " pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.637066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.637042    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:52:27.638589 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:27.638563    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1c088d_4fdf_4f6c_92f4_40f5809564a6.slice/crio-fa58e043925eac9b1b9b2c4aaae3b637dc8642e36ac7ef6cd259fd3ee9a3bc9f WatchSource:0}: Error finding container fa58e043925eac9b1b9b2c4aaae3b637dc8642e36ac7ef6cd259fd3ee9a3bc9f: Status 404 returned error can't find the container with id fa58e043925eac9b1b9b2c4aaae3b637dc8642e36ac7ef6cd259fd3ee9a3bc9f
Oct 30 02:52:27.685427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.685399    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:27.988718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:27.988692    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74685b6749-zmk6j"]
Oct 30 02:52:27.990581 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:27.990558    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0088fc29_fd63_4aa4_b655_c8482f25fbdb.slice/crio-1fa10e819afc4e76f5affb4f3c17fb1dfa96e00516044d3586629214d5a43113 WatchSource:0}: Error finding container 1fa10e819afc4e76f5affb4f3c17fb1dfa96e00516044d3586629214d5a43113: Status 404 returned error can't find the container with id 1fa10e819afc4e76f5affb4f3c17fb1dfa96e00516044d3586629214d5a43113
Oct 30 02:52:28.106539 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:28.106495    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" event={"ID":"dd1c088d-4fdf-4f6c-92f4-40f5809564a6","Type":"ContainerStarted","Data":"fa58e043925eac9b1b9b2c4aaae3b637dc8642e36ac7ef6cd259fd3ee9a3bc9f"}
Oct 30 02:52:28.107895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:28.107861    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j" event={"ID":"0088fc29-fd63-4aa4-b655-c8482f25fbdb","Type":"ContainerStarted","Data":"1fa10e819afc4e76f5affb4f3c17fb1dfa96e00516044d3586629214d5a43113"}
Oct 30 02:52:29.115297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:29.115259    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j" event={"ID":"0088fc29-fd63-4aa4-b655-c8482f25fbdb","Type":"ContainerStarted","Data":"32a26e6824f7c7511b429c501cf26e8733481aacf34264147be42f6664724b5c"}
Oct 30 02:52:29.115718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:29.115322    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:30.044549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.044518    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:30.044650 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.044588    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:30.044650 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.044621    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:30.122100 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.122060    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" event={"ID":"dd1c088d-4fdf-4f6c-92f4-40f5809564a6","Type":"ContainerStarted","Data":"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"}
Oct 30 02:52:30.181329 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.181274    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podStartSLOduration=1.7789469470000001 podStartE2EDuration="4.181259097s" totalImagesPullingTime="2.40231215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:30.180396405 +0000 UTC m=+689.254567198" watchObservedRunningTime="2026-10-30 02:52:30.181259097 +0000 UTC m=+689.255429890"
Oct 30 02:52:30.181826 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.181793    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j" podStartSLOduration=2.857546498 podStartE2EDuration="3.181782489s" totalImagesPullingTime="324.235991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:29.169993853 +0000 UTC m=+688.244164637" watchObservedRunningTime="2026-10-30 02:52:30.181782489 +0000 UTC m=+689.255953313"
Oct 30 02:52:30.297420 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:30.297388    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:31.073844 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:31.073815    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8tfqx"
Oct 30 02:52:31.298332 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:31.298294    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:31.298749 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:31.298360    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:32.296851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:32.296819    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:32.297041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:32.296870    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:33.296940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:33.296905    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:33.297378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:33.296961    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:34.297020 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:34.296975    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:34.297420 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:34.297037    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:35.297712 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:35.297674    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:35.298116 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:35.297730    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:36.296753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:36.296720    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:36.296925 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:36.296778    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:37.297455 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:37.297417    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:52:37.297830 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:37.297479    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 02:52:38.301111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:38.301081    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:38.301545 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:38.301463    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:38.302034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:38.302017    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:52:39.163260 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:52:39.163203    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:52:40.124033 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.124004    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74685b6749-zmk6j"
Oct 30 02:52:40.231442 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.231414    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:40.231611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.231588    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" podUID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" containerName="manager" containerID="cri-o://3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e" gracePeriod=30
Oct 30 02:52:40.482342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.482316    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:40.533141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.533116    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\") pod \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") "
Oct 30 02:52:40.533260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.533147    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\" (UniqueName: \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\") pod \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\" (UID: \"9fe482e9-b27d-4cf6-a01f-5601aef5409f\") "
Oct 30 02:52:40.535184 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.535151    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs" pod "9fe482e9-b27d-4cf6-a01f-5601aef5409f" (UID: "9fe482e9-b27d-4cf6-a01f-5601aef5409f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:52:40.535321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.535271    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr" pod "9fe482e9-b27d-4cf6-a01f-5601aef5409f" (UID: "9fe482e9-b27d-4cf6-a01f-5601aef5409f"). InnerVolumeSpecName "kube-api-access-96sdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:52:40.634160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.634130    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe482e9-b27d-4cf6-a01f-5601aef5409f-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:52:40.634160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:40.634154    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96sdr\" (UniqueName: \"kubernetes.io/projected/9fe482e9-b27d-4cf6-a01f-5601aef5409f-kube-api-access-96sdr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:52:41.209627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.209592    4214 generic.go:356] "Generic (PLEG): container finished" podID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" containerID="3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e" exitCode=0
Oct 30 02:52:41.210117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.209649    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"
Oct 30 02:52:41.210117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.209672    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" event={"ID":"9fe482e9-b27d-4cf6-a01f-5601aef5409f","Type":"ContainerDied","Data":"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"}
Oct 30 02:52:41.210117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.209723    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-7np9b" event={"ID":"9fe482e9-b27d-4cf6-a01f-5601aef5409f","Type":"ContainerDied","Data":"5edbf3ab4952d882e73151b0d92ece745a2ef4a3b3be4d5f2ff9796d8b05b2b4"}
Oct 30 02:52:41.210117 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.209745    4214 scope.go:122] "RemoveContainer" containerID="3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"
Oct 30 02:52:41.218419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.218400    4214 scope.go:122] "RemoveContainer" containerID="3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"
Oct 30 02:52:41.218691 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:52:41.218670    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e\": container with ID starting with 3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e not found: ID does not exist" containerID="3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"
Oct 30 02:52:41.218740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.218702    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e"} err="failed to get container status \"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e\": rpc error: code = NotFound desc = could not find container \"3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e\": container with ID starting with 3a2f6543c1f2d5a9a55acab925e5bffec948acfb5b27bd4a2115e7be25a1050e not found: ID does not exist"
Oct 30 02:52:41.264700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.264674    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:41.277594 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:41.277568    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-7np9b"]
Oct 30 02:52:42.026131 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.026088    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:52:42.026867 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.026846    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" containerName="manager"
Oct 30 02:52:42.026973 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.026868    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" containerName="manager"
Oct 30 02:52:42.027100 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.027085    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" containerName="manager"
Oct 30 02:52:42.032115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.032077    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.051578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.051554    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:52:42.147390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147354    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147413    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147613 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147573    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147632    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f25h\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147699    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147880 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147861    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.147943 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147893    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.148002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.147975    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.148068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.148050    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249070    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249100    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f25h\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249119    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249141    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249157    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249178    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249203    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249262    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249293    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249557    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249611    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249648    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.249878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249736    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.250015 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.249995    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.251438 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.251417    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.251669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.251651    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.269451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.269426    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.269677 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.269658    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f25h\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-q5gc5\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.345601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.345571    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:42.658963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.658882    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:52:42.665668 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:52:42.665640    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c5622_fdb2_4ba2_b18c_f62ae4dd42d6.slice/crio-60c166b7461d23e3fcd0f3da94b1460bb433032ab66a9009ae6841fbc7ba7635 WatchSource:0}: Error finding container 60c166b7461d23e3fcd0f3da94b1460bb433032ab66a9009ae6841fbc7ba7635: Status 404 returned error can't find the container with id 60c166b7461d23e3fcd0f3da94b1460bb433032ab66a9009ae6841fbc7ba7635
Oct 30 02:52:42.667883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.667857    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:42.667950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.667922    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:42.667999 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:42.667951    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:52:43.065687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.065655    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe482e9-b27d-4cf6-a01f-5601aef5409f" path="/var/lib/kubelet/pods/9fe482e9-b27d-4cf6-a01f-5601aef5409f/volumes"
Oct 30 02:52:43.222862 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.222826    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" event={"ID":"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6","Type":"ContainerStarted","Data":"097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26"}
Oct 30 02:52:43.223075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.222889    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" event={"ID":"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6","Type":"ContainerStarted","Data":"60c166b7461d23e3fcd0f3da94b1460bb433032ab66a9009ae6841fbc7ba7635"}
Oct 30 02:52:43.303908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.303838    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podStartSLOduration=2.303822994 podStartE2EDuration="2.303822994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:43.289869865 +0000 UTC m=+702.364040659" watchObservedRunningTime="2026-10-30 02:52:43.303822994 +0000 UTC m=+702.377993788"
Oct 30 02:52:43.353397 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.353302    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:43.358709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.358357    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:43.358709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:43.358425    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:44.351601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:44.351565    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:44.352164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:44.352131    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:45.352252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:45.348558    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:45.352252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:45.348615    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:46.348615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:46.348578    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:46.348916 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:46.348874    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:47.353614 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:47.353578    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:47.354182 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:47.354153    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:48.354253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:48.351553    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:48.354253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:48.351624    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:49.346529 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:49.346494    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 02:52:49.346713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:49.346554    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 02:52:50.891824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:50.891790    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:50.892325 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:50.892070    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:52:50.893304 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:52:50.893282    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:53:09.163571 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:53:09.163532    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:53:13.255796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.255764    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 02:53:13.268393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.268369    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.282590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.282560    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 02:53:13.290682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.290658    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d9f3ac-openshift-default-dockercfg-97t95\""
Oct 30 02:53:13.349177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.348917    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349090    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349128    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349288    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349324    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349351    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44k6f\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349418    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349618 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349452    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.349618 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.349554    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450465 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450435    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450494    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450518    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450556    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450574    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.450911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450888    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451005 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450888    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44k6f\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451005 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450954    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451005 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.450981    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.451012    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.451077    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.451285    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.451387    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.451850 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.451825    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.452746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.452720    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.453056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.453037    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.473396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.473374    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.478047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.478030    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44k6f\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.584132 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.584103    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:13.612734 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.612701    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 02:53:13.616969 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.616952    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.651616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.651539    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-aeff1d-openshift-default-dockercfg-8cpz8\""
Oct 30 02:53:13.670014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.668379    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 02:53:13.755034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.754988    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755215 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755143    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755215 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755206    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755312    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755346    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755418    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755454    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755528    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5f2\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.755573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.755570    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.856818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856735    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.856818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856793    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856849    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856876    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856915    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856948    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5f2\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.856987    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.857043    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.857210    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.857367    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.857417    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.857514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.857435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.858028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.858001    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.859103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.859082    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.859579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.859562    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.877087 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.877043    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.877448 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.877431    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5f2\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.915640 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.915170    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 02:53:13.920955 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.920908    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:13.921072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.920983    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:13.921072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.921026    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:13.931549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.931471    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 02:53:13.931668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.931585    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:13.933165 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.933141    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:13.943587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.943564    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 02:53:13.948964 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:13.948930    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7d1788-openshift-default-dockercfg-29c4p\""
Oct 30 02:53:14.059488 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059403    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059488 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059455    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059619    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059668    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdbn\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059759    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059821    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.059859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059849    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.060014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059898    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.060014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.059923    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.160627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160545    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdbn\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.160627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160617    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.160841 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160676    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.160841 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160704    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.160952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160927    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.161012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.160969    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.161070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161010    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.161070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161051    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.161177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161084    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.161177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161149    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.162071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.162071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.161683    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.162071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.162034    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.162315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.162076    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.163641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.163603    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.164602 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.164579    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.197378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.197355    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.197538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.197522    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdbn\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.264632 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.258545    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:14.308611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.308578    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 02:53:14.320546 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:14.320186    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432ca094_9b1c_4d92_9223_91f93b9bb441.slice/crio-364df988e722d9e2c4840f779b705788111bb5a6f6255061e3b42b771c5847b5 WatchSource:0}: Error finding container 364df988e722d9e2c4840f779b705788111bb5a6f6255061e3b42b771c5847b5: Status 404 returned error can't find the container with id 364df988e722d9e2c4840f779b705788111bb5a6f6255061e3b42b771c5847b5
Oct 30 02:53:14.325391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.323207    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:14.325391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.323327    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:14.325391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.323374    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:14.413405 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.413343    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerStarted","Data":"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"}
Oct 30 02:53:14.413405 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.413395    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerStarted","Data":"ac67e4fe412dec98136cbb265ae70d9ff8b3015c792c548572f57f0361e762d7"}
Oct 30 02:53:14.415396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.415373    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerStarted","Data":"364df988e722d9e2c4840f779b705788111bb5a6f6255061e3b42b771c5847b5"}
Oct 30 02:53:14.463788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.463729    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podStartSLOduration=1.4637087659999999 podStartE2EDuration="1.463708766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:14.461531977 +0000 UTC m=+733.535702771" watchObservedRunningTime="2026-10-30 02:53:14.463708766 +0000 UTC m=+733.537879562"
Oct 30 02:53:14.591118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.590995    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:14.597786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.597753    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:14.617477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.617203    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 02:53:14.618205 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:14.618179    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41471dd_1cb4_473c_8615_bc63682802dc.slice/crio-743f111e0285108192b0efa70db393741df620d4586f7bec5a14de37cf571006 WatchSource:0}: Error finding container 743f111e0285108192b0efa70db393741df620d4586f7bec5a14de37cf571006: Status 404 returned error can't find the container with id 743f111e0285108192b0efa70db393741df620d4586f7bec5a14de37cf571006
Oct 30 02:53:14.621552 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.621516    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:14.621664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.621598    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:14.621664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:14.621643    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:15.422751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.422713    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerStarted","Data":"3eb0fa0927ea5589edaa431107642a542ca14f919c4aba398859b4e9a43e8ce0"}
Oct 30 02:53:15.423267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.422772    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerStarted","Data":"743f111e0285108192b0efa70db393741df620d4586f7bec5a14de37cf571006"}
Oct 30 02:53:15.424633 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.424604    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerStarted","Data":"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"}
Oct 30 02:53:15.424777 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.424763    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:15.425751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.425732    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:15.493308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.493256    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podStartSLOduration=2.493242891 podStartE2EDuration="2.493242891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:15.479496258 +0000 UTC m=+734.553667053" watchObservedRunningTime="2026-10-30 02:53:15.493242891 +0000 UTC m=+734.567413675"
Oct 30 02:53:15.572951 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.572899    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podStartSLOduration=2.572883746 podStartE2EDuration="2.572883746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:15.571414277 +0000 UTC m=+734.645585082" watchObservedRunningTime="2026-10-30 02:53:15.572883746 +0000 UTC m=+734.647054539"
Oct 30 02:53:15.934573 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.934539    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:15.939391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:15.939365    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:16.260352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.260268    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:16.265322 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.265297    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:16.431416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.431389    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:16.432187 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.431437    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:16.432530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.432507    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:16.432629 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:16.432585    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:19.006203 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.006163    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 02:53:19.009893 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.009870    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.015760 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.015736    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 02:53:19.019160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.019142    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.019642 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.019618    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.019698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.019668    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlz4\" (UniqueName: \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.019795 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.019780    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgzq\" (UniqueName: \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.019914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.019898    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.033479 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.032959    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-32bad3\""
Oct 30 02:53:19.036511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.036485    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 02:53:19.044975 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.044950    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-32bad3-dockercfg-f6lzc\""
Oct 30 02:53:19.051262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.051241    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 02:53:19.147827 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.147780    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlz4\" (UniqueName: \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.149217 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.149181    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgzq\" (UniqueName: \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.149531 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.149509    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.155246 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.149685    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.155431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.150523    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.155515 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.154077    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.182037 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.182008    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgzq\" (UniqueName: \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\") pod \"payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.183527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.183501    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlz4\" (UniqueName: \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\") pod \"payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.325837 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.325803    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:19.334035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.334010    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:19.695500 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.695477    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 02:53:19.697780 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:19.697718    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba075b4_a86a_4c49_adfd_5063f81a084e.slice/crio-0b7536e8c969ee7ab84aa3972d452df5465d14c2bb1a7fa27f6c08a991ef8887 WatchSource:0}: Error finding container 0b7536e8c969ee7ab84aa3972d452df5465d14c2bb1a7fa27f6c08a991ef8887: Status 404 returned error can't find the container with id 0b7536e8c969ee7ab84aa3972d452df5465d14c2bb1a7fa27f6c08a991ef8887
Oct 30 02:53:19.745375 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:19.745345    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 02:53:19.761628 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:19.761599    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb830bf8a_554a_4597_ad34_9785136e7044.slice/crio-63a9b9462affcb0669de9bbdfb8347d8cb3374fe1294cc5a5cd749bf79136f93 WatchSource:0}: Error finding container 63a9b9462affcb0669de9bbdfb8347d8cb3374fe1294cc5a5cd749bf79136f93: Status 404 returned error can't find the container with id 63a9b9462affcb0669de9bbdfb8347d8cb3374fe1294cc5a5cd749bf79136f93
Oct 30 02:53:20.457811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.457768    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" event={"ID":"b830bf8a-554a-4597-ad34-9785136e7044","Type":"ContainerStarted","Data":"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"}
Oct 30 02:53:20.458336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.457836    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:20.458336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.457858    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" event={"ID":"b830bf8a-554a-4597-ad34-9785136e7044","Type":"ContainerStarted","Data":"63a9b9462affcb0669de9bbdfb8347d8cb3374fe1294cc5a5cd749bf79136f93"}
Oct 30 02:53:20.459590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.459566    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" event={"ID":"7ba075b4-a86a-4c49-adfd-5063f81a084e","Type":"ContainerStarted","Data":"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"}
Oct 30 02:53:20.459693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.459607    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" event={"ID":"7ba075b4-a86a-4c49-adfd-5063f81a084e","Type":"ContainerStarted","Data":"0b7536e8c969ee7ab84aa3972d452df5465d14c2bb1a7fa27f6c08a991ef8887"}
Oct 30 02:53:20.459693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.459627    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:20.546297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.546244    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" podStartSLOduration=2.546215652 podStartE2EDuration="2.546215652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:20.537776834 +0000 UTC m=+739.611947626" watchObservedRunningTime="2026-10-30 02:53:20.546215652 +0000 UTC m=+739.620386445"
Oct 30 02:53:20.650886 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:20.650627    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" podStartSLOduration=2.650610443 podStartE2EDuration="2.650610443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:20.650243602 +0000 UTC m=+739.724414386" watchObservedRunningTime="2026-10-30 02:53:20.650610443 +0000 UTC m=+739.724781237"
Oct 30 02:53:21.755807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.755775    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 02:53:21.804786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.802610    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 02:53:21.804786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.802754    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:21.838325 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.838304    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d9f3ac-dockercfg-gfgcv\""
Oct 30 02:53:21.842840 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.842815    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d9f3ac\""
Oct 30 02:53:21.886395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.886359    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzjq\" (UniqueName: \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:21.886538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.886476    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:21.988278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.988248    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzjq\" (UniqueName: \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:21.988440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.988300    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:21.988963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:21.988945    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:22.016487 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:22.016425    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzjq\" (UniqueName: \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\") pod \"payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:22.118270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:22.118214    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:22.436602 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:22.436569    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 02:53:22.437690 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:22.437666    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18381b7_5795_4c53_954c_33859dee342b.slice/crio-525fcf557522f1a810c2dce5b5150a1fd1fe2a9986ec4983c269a07b151724d0 WatchSource:0}: Error finding container 525fcf557522f1a810c2dce5b5150a1fd1fe2a9986ec4983c269a07b151724d0: Status 404 returned error can't find the container with id 525fcf557522f1a810c2dce5b5150a1fd1fe2a9986ec4983c269a07b151724d0
Oct 30 02:53:22.474593 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:22.472768    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" event={"ID":"d18381b7-5795-4c53-954c-33859dee342b","Type":"ContainerStarted","Data":"525fcf557522f1a810c2dce5b5150a1fd1fe2a9986ec4983c269a07b151724d0"}
Oct 30 02:53:23.482896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.482864    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" event={"ID":"d18381b7-5795-4c53-954c-33859dee342b","Type":"ContainerStarted","Data":"c86ac29a4220eeb133f4054dc02eea42271b084aa4f07230442fb4b498f6bb6f"}
Oct 30 02:53:23.483447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.483422    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:23.553353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.553305    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" podStartSLOduration=2.553293417 podStartE2EDuration="2.553293417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:23.550364238 +0000 UTC m=+742.624535030" watchObservedRunningTime="2026-10-30 02:53:23.553293417 +0000 UTC m=+742.627464209"
Oct 30 02:53:23.681104 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.681063    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 02:53:23.684859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.684837    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.705181 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.705148    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.705409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.705221    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-f7d1a1-dockercfg-q85fm\""
Oct 30 02:53:23.705409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.705317    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdspp\" (UniqueName: \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.705532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.705508    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-f7d1a1\""
Oct 30 02:53:23.707036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.707008    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 02:53:23.806435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.806389    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdspp\" (UniqueName: \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.807189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.807163    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.808090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.808065    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.831467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.831430    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 02:53:23.835537 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.835504    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:23.857993 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.857964    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdspp\" (UniqueName: \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\") pod \"payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:23.870790 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.870762    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 02:53:23.908309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.908279    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:23.908436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.908414    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f89\" (UniqueName: \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:23.997613 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:23.997581    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:24.009399 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.009374    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f89\" (UniqueName: \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:24.009497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.009473    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:24.010275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.010248    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:24.036751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.036720    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f89\" (UniqueName: \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\") pod \"payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:24.152041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.152009    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:24.305754 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.305708    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 02:53:24.314558 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:24.312380    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fb0d90_45b4_454f_95f3_ae7c84e2448f.slice/crio-3e4e9697994f63277e123750dc9079510236d39d50433fb016de525e9915f419 WatchSource:0}: Error finding container 3e4e9697994f63277e123750dc9079510236d39d50433fb016de525e9915f419: Status 404 returned error can't find the container with id 3e4e9697994f63277e123750dc9079510236d39d50433fb016de525e9915f419
Oct 30 02:53:24.457824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.457797    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 02:53:24.459088 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:24.459060    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61881be2_60e7_480b_8ab8_40544091d90e.slice/crio-26054591336b01faee737e70727a61b83d405376e229c08e7660879ad97fad41 WatchSource:0}: Error finding container 26054591336b01faee737e70727a61b83d405376e229c08e7660879ad97fad41: Status 404 returned error can't find the container with id 26054591336b01faee737e70727a61b83d405376e229c08e7660879ad97fad41
Oct 30 02:53:24.490431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.490397    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" event={"ID":"61881be2-60e7-480b-8ab8-40544091d90e","Type":"ContainerStarted","Data":"26054591336b01faee737e70727a61b83d405376e229c08e7660879ad97fad41"}
Oct 30 02:53:24.492199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.492173    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" event={"ID":"41fb0d90-45b4-454f-95f3-ae7c84e2448f","Type":"ContainerStarted","Data":"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"}
Oct 30 02:53:24.492353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.492254    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" event={"ID":"41fb0d90-45b4-454f-95f3-ae7c84e2448f","Type":"ContainerStarted","Data":"3e4e9697994f63277e123750dc9079510236d39d50433fb016de525e9915f419"}
Oct 30 02:53:24.492436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.492416    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:24.535980 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:24.535940    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" podStartSLOduration=1.53592689 podStartE2EDuration="1.53592689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:24.53382641 +0000 UTC m=+743.607997203" watchObservedRunningTime="2026-10-30 02:53:24.53592689 +0000 UTC m=+743.610097683"
Oct 30 02:53:25.499717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.499677    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" event={"ID":"61881be2-60e7-480b-8ab8-40544091d90e","Type":"ContainerStarted","Data":"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"}
Oct 30 02:53:25.500191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.499835    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:25.544262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.544195    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" podStartSLOduration=2.544176984 podStartE2EDuration="2.544176984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:25.540965881 +0000 UTC m=+744.615136675" watchObservedRunningTime="2026-10-30 02:53:25.544176984 +0000 UTC m=+744.618347810"
Oct 30 02:53:25.618725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.618695    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 02:53:25.622761 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.622737    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.636206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.636182    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 02:53:25.640163 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.640137    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-810119\""
Oct 30 02:53:25.640326 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.640283    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-810119-dockercfg-bbzgw\""
Oct 30 02:53:25.732017 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.731972    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mb4\" (UniqueName: \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.732194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.732028    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.833093 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.833054    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mb4\" (UniqueName: \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.833093 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.833094    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.833877 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.833857    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.871033 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.870998    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mb4\" (UniqueName: \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\") pod \"payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:25.943193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:25.943153    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:26.277190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.277165    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 02:53:26.278934 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:26.278898    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324658d1_42c7_4fba_90a7_7ff7b23de35c.slice/crio-917956f19d4477642de21d9abbe43f1cae791087b06b9481a6de49fa13a47413 WatchSource:0}: Error finding container 917956f19d4477642de21d9abbe43f1cae791087b06b9481a6de49fa13a47413: Status 404 returned error can't find the container with id 917956f19d4477642de21d9abbe43f1cae791087b06b9481a6de49fa13a47413
Oct 30 02:53:26.511220 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.511188    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" event={"ID":"324658d1-42c7-4fba-90a7-7ff7b23de35c","Type":"ContainerStarted","Data":"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"}
Oct 30 02:53:26.511220 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.511252    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" event={"ID":"324658d1-42c7-4fba-90a7-7ff7b23de35c","Type":"ContainerStarted","Data":"917956f19d4477642de21d9abbe43f1cae791087b06b9481a6de49fa13a47413"}
Oct 30 02:53:26.564979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.563588    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" podStartSLOduration=1.563572768 podStartE2EDuration="1.563572768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:26.563452377 +0000 UTC m=+745.637623163" watchObservedRunningTime="2026-10-30 02:53:26.563572768 +0000 UTC m=+745.637743562"
Oct 30 02:53:26.746623 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.746589    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 02:53:26.750364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.750350    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.777288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.777150    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e11f03-dockercfg-jk7rq\""
Oct 30 02:53:26.777288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.777262    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e11f03\""
Oct 30 02:53:26.779789 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.779765    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 02:53:26.850359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.847689    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqpx\" (UniqueName: \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.850359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.847799    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.871435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.871399    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 02:53:26.876869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.876827    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 02:53:26.877029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.876975    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:26.948534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.948501    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqpx\" (UniqueName: \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.948700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.948598    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.948700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.948641    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:26.948819 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.948710    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9h6x\" (UniqueName: \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:26.949346 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.949327    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:26.968905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:26.968876    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqpx\" (UniqueName: \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\") pod \"payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:27.049247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.049198    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:27.049381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.049287    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9h6x\" (UniqueName: \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:27.049825 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.049807    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:27.063693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.063670    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:27.070185 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.070161    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9h6x\" (UniqueName: \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\") pod \"payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:27.192577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.192539    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:27.378739 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.378712    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 02:53:27.380205 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:27.380179    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c9e5d7_8358_4041_93ea_5d9cc1d1f226.slice/crio-6386b88a73a8fe1c739c7a5e0e9dde5de13d98e7fb98eca1b75afb511857a6cc WatchSource:0}: Error finding container 6386b88a73a8fe1c739c7a5e0e9dde5de13d98e7fb98eca1b75afb511857a6cc: Status 404 returned error can't find the container with id 6386b88a73a8fe1c739c7a5e0e9dde5de13d98e7fb98eca1b75afb511857a6cc
Oct 30 02:53:27.519651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.519620    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" event={"ID":"36c9e5d7-8358-4041-93ea-5d9cc1d1f226","Type":"ContainerStarted","Data":"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"}
Oct 30 02:53:27.519995 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.519681    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" event={"ID":"36c9e5d7-8358-4041-93ea-5d9cc1d1f226","Type":"ContainerStarted","Data":"6386b88a73a8fe1c739c7a5e0e9dde5de13d98e7fb98eca1b75afb511857a6cc"}
Oct 30 02:53:27.520165 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.520148    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:27.520254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.520186    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 02:53:27.520254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.520210    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:27.521205 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:27.521179    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2618539f_c69f_43ac_8db3_30704cfaf7cb.slice/crio-ba2b5a5c47e02e3b5aa813fdf1af6d6bb9e10bb828229b18931336eb67c49692 WatchSource:0}: Error finding container ba2b5a5c47e02e3b5aa813fdf1af6d6bb9e10bb828229b18931336eb67c49692: Status 404 returned error can't find the container with id ba2b5a5c47e02e3b5aa813fdf1af6d6bb9e10bb828229b18931336eb67c49692
Oct 30 02:53:27.577381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:27.577336    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" podStartSLOduration=1.5773241580000001 podStartE2EDuration="1.577324158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:27.574284321 +0000 UTC m=+746.648455117" watchObservedRunningTime="2026-10-30 02:53:27.577324158 +0000 UTC m=+746.651494951"
Oct 30 02:53:28.528469 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.528428    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" event={"ID":"2618539f-c69f-43ac-8db3-30704cfaf7cb","Type":"ContainerStarted","Data":"62646af75d121d0b6a54dae1b79a1f92671de95c8c960fbf2ec741631937eb34"}
Oct 30 02:53:28.528911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.528800    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:28.528911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.528834    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" event={"ID":"2618539f-c69f-43ac-8db3-30704cfaf7cb","Type":"ContainerStarted","Data":"ba2b5a5c47e02e3b5aa813fdf1af6d6bb9e10bb828229b18931336eb67c49692"}
Oct 30 02:53:28.541537 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.541512    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 02:53:28.545328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.545304    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.570297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.570272    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-aeff1d\""
Oct 30 02:53:28.570786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.570756    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbck\" (UniqueName: \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.570970 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.570949    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.575615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.575598    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-aeff1d-dockercfg-5fzvm\""
Oct 30 02:53:28.587863 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.587838    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 02:53:28.611816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.611752    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" podStartSLOduration=2.611734165 podStartE2EDuration="2.611734165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:28.607864676 +0000 UTC m=+747.682035474" watchObservedRunningTime="2026-10-30 02:53:28.611734165 +0000 UTC m=+747.685904960"
Oct 30 02:53:28.672437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.671962    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.672437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.672103    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbck\" (UniqueName: \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.673048 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.672982    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.698697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.698673    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbck\" (UniqueName: \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\") pod \"payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:28.859791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:28.859694    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:29.169579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.168747    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 02:53:29.171548 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:29.171522    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df033e2_92e3_40b1_9552_ae4fe56cd446.slice/crio-a446bacc5f54591204b6e915cfd05188d33e863da95333c6846f37980ead4a28 WatchSource:0}: Error finding container a446bacc5f54591204b6e915cfd05188d33e863da95333c6846f37980ead4a28: Status 404 returned error can't find the container with id a446bacc5f54591204b6e915cfd05188d33e863da95333c6846f37980ead4a28
Oct 30 02:53:29.570331 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.570179    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" event={"ID":"1df033e2-92e3-40b1-9552-ae4fe56cd446","Type":"ContainerStarted","Data":"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"}
Oct 30 02:53:29.571123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.571064    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" event={"ID":"1df033e2-92e3-40b1-9552-ae4fe56cd446","Type":"ContainerStarted","Data":"a446bacc5f54591204b6e915cfd05188d33e863da95333c6846f37980ead4a28"}
Oct 30 02:53:29.571411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.571379    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:29.618998 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.618946    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" podStartSLOduration=1.61893185 podStartE2EDuration="1.61893185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:29.617482611 +0000 UTC m=+748.691653404" watchObservedRunningTime="2026-10-30 02:53:29.61893185 +0000 UTC m=+748.693102642"
Oct 30 02:53:29.826169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.826079    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 02:53:29.830078 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.830055    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:29.883077 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.883045    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7d1788\""
Oct 30 02:53:29.884358 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.883605    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7d1788-dockercfg-ct5td\""
Oct 30 02:53:29.889563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.889535    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 02:53:29.986180 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.986149    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:29.986378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:29.986258    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bcx\" (UniqueName: \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.087948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.087862    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.088216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.088191    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bcx\" (UniqueName: \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.088771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.088713    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.110209 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.110170    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bcx\" (UniqueName: \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\") pod \"payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.195413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.195376    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:30.561431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.561390    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 02:53:30.566355 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:53:30.566325    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d41e703_823a_4555_8aef_87719d078e7d.slice/crio-3eb51ee48eb7c6856395176542aaa721943e428a180a810f915e8cd11b48ec29 WatchSource:0}: Error finding container 3eb51ee48eb7c6856395176542aaa721943e428a180a810f915e8cd11b48ec29: Status 404 returned error can't find the container with id 3eb51ee48eb7c6856395176542aaa721943e428a180a810f915e8cd11b48ec29
Oct 30 02:53:30.577327 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:30.577294    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" event={"ID":"3d41e703-823a-4555-8aef-87719d078e7d","Type":"ContainerStarted","Data":"3eb51ee48eb7c6856395176542aaa721943e428a180a810f915e8cd11b48ec29"}
Oct 30 02:53:31.468278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:31.468223    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 02:53:31.468475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:31.468407    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 02:53:31.585015 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:31.584973    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" event={"ID":"3d41e703-823a-4555-8aef-87719d078e7d","Type":"ContainerStarted","Data":"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"}
Oct 30 02:53:31.585517 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:31.585050    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:31.675174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:31.674973    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" podStartSLOduration=2.6749509849999997 podStartE2EDuration="2.674950985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:31.672489194 +0000 UTC m=+750.746659988" watchObservedRunningTime="2026-10-30 02:53:31.674950985 +0000 UTC m=+750.749121759"
Oct 30 02:53:34.500488 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:34.500451    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 02:53:35.125038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.124695    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 02:53:35.129123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.129101    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.175253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.172470    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-f7d1a1-serving-cert\""
Oct 30 02:53:35.193255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.191746    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bzc5m\""
Oct 30 02:53:35.193255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.191879    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 02:53:35.200360 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.200274    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 02:53:35.202049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.202023    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxj6s\" (UniqueName: \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.202195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.202142    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.202195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.202179    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.202334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.202291    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.302988 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.302954    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.302988 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.302992    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.303282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.303058    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.303282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.303114    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxj6s\" (UniqueName: \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.304163 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.304135    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.307363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.307337    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.308269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.308215    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.370305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.358947    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxj6s\" (UniqueName: \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\") pod \"maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.503294 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.503181    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:35.504590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.504569    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 02:53:35.896759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:35.896575    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 02:53:36.515311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:36.515281    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 02:53:36.633567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:36.633487    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" event={"ID":"c40f86a2-bcfa-41f4-a00c-433ebea44c71","Type":"ContainerStarted","Data":"a34da35f540223e802a6ba9dbf2f39b28c09481c6c88bce0e8bd6a722ea28077"}
Oct 30 02:53:38.533288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:38.533259    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 02:53:39.223927 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:53:39.223829    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:53:39.587715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:39.587044    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 02:53:39.587715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:39.587465    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 02:53:40.585933 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:40.585893    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 02:53:40.748260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:40.744957    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" event={"ID":"c40f86a2-bcfa-41f4-a00c-433ebea44c71","Type":"ContainerStarted","Data":"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"}
Oct 30 02:53:40.748260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:40.745914    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:40.792371 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:40.792302    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" podStartSLOduration=1.78299527 podStartE2EDuration="5.792282297s" totalImagesPullingTime="4.009287027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:40.79073142 +0000 UTC m=+759.864902253" watchObservedRunningTime="2026-10-30 02:53:40.792282297 +0000 UTC m=+759.866453091"
Oct 30 02:53:41.754652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.754571    4214 generic.go:356] "Generic (PLEG): container finished" podID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerID="e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b" exitCode=0
Oct 30 02:53:41.755180 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.754650    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerDied","Data":"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"}
Oct 30 02:53:41.755282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.755215    4214 scope.go:122] "RemoveContainer" containerID="e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"
Oct 30 02:53:41.756532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.756505    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:41.756622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.756600    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:41.756682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:41.756633    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:42.593857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:42.593826    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 02:53:42.798175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:42.798145    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerStarted","Data":"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"}
Oct 30 02:53:42.934047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:42.933955    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:42.941696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:42.941362    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:42.941696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:42.941422    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:43.806508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.806479    4214 generic.go:356] "Generic (PLEG): container finished" podID="79f816da-469c-44b8-8431-73198aa7ecde" containerID="068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31" exitCode=0
Oct 30 02:53:43.807657 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.806762    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerDied","Data":"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"}
Oct 30 02:53:43.807657 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.807141    4214 scope.go:122] "RemoveContainer" containerID="068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"
Oct 30 02:53:43.808518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.808493    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:43.808607 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.808580    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:43.808646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.808634    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:43.938250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.934408    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:43.938250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:43.934465    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:44.815551 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.815518    4214 generic.go:356] "Generic (PLEG): container finished" podID="c41471dd-1cb4-473c-8615-bc63682802dc" containerID="3eb0fa0927ea5589edaa431107642a542ca14f919c4aba398859b4e9a43e8ce0" exitCode=0
Oct 30 02:53:44.816053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.815594    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerDied","Data":"3eb0fa0927ea5589edaa431107642a542ca14f919c4aba398859b4e9a43e8ce0"}
Oct 30 02:53:44.816123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.816066    4214 scope.go:122] "RemoveContainer" containerID="3eb0fa0927ea5589edaa431107642a542ca14f919c4aba398859b4e9a43e8ce0"
Oct 30 02:53:44.817514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.817486    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:44.817614 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.817554    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:44.817614 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.817591    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:53:44.817805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.817789    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerStarted","Data":"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"}
Oct 30 02:53:44.938378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.938347    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:44.938642 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:44.938612    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:45.425366 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.425326    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:45.585595 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.585563    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:45.589929 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.589897    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 02:53:45.590205 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.590181    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 02:53:45.829759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.829704    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerStarted","Data":"4c3c289cce4e6d30ea13825703128e3e3b6ec1405cf89d523bcc15ede0bc4a5c"}
Oct 30 02:53:45.933996 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.933952    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:45.934146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:45.934015    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:46.262310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.262042    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:46.262310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.262146    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:46.262310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.262293    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:46.431502 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.431355    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:46.431502 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.431417    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:46.590640 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.590602    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 02:53:46.590937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.590906    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 02:53:46.934023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.933948    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:46.934549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:46.934520    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:47.261310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.260599    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:47.261310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.260662    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:47.584671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.584635    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 02:53:47.584988 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.584959    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 02:53:47.829088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.829058    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 02:53:47.978350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.970644    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 02:53:47.978350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:47.970722    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 02:53:48.260793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.260697    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:48.261075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.261022    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:48.585263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.584633    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 02:53:48.585263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.584696    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 02:53:48.957601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.957516    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:48.967686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:48.967649    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 02:53:49.274695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:49.274600    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:49.275283 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:49.275251    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:49.590253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:49.589326    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 02:53:49.590253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:49.589383    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 02:53:50.260689 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:50.260648    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:50.260985 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:50.260956    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:50.717509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:50.717470    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:50.720100 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:50.720076    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 02:53:51.260164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:51.260130    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 02:53:51.260418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:51.260184    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 02:53:52.264618 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:52.264585    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:53:52.265647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:53:52.265626    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 02:54:09.165984 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:54:09.165937    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:54:20.362476 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.362444    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 02:54:20.366629 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.366597    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381604 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381569    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381618    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381647    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4kv\" (UniqueName: \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381814    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381925 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381872    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.381925 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.381913    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.387330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.387299    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 02:54:20.395527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.395500    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"kube-root-ca.crt\""
Oct 30 02:54:20.397986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.397967    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"default-dockercfg-g2vqc\""
Oct 30 02:54:20.397986 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.397978    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"openshift-service-ca.crt\""
Oct 30 02:54:20.398187 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.398119    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"premium-w6-32bad3-kserve-self-signed-certs\""
Oct 30 02:54:20.483014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.482978    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483201 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483035    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483201 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483075    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483201 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483179    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483208    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483267    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4kv\" (UniqueName: \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483673 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483642    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483673 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483656    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.483852 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.483800    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.485333 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.485313    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.485801 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.485783    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.503048 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.503012    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4kv\" (UniqueName: \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\") pod \"premium-w6-32bad3-kserve-5c8c79f499-n6d6t\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") " pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.681324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.681215    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:20.989372 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:20.989343    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 02:54:20.990745 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:54:20.990718    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a18368_7423_47c1_b0dc_c2aaf3999a42.slice/crio-18c249b6beba35e8d303c2c545751ba5be02d6808c1fa245fea1185e2d8ff8af WatchSource:0}: Error finding container 18c249b6beba35e8d303c2c545751ba5be02d6808c1fa245fea1185e2d8ff8af: Status 404 returned error can't find the container with id 18c249b6beba35e8d303c2c545751ba5be02d6808c1fa245fea1185e2d8ff8af
Oct 30 02:54:21.179213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:21.179165    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" event={"ID":"93a18368-7423-47c1-b0dc-c2aaf3999a42","Type":"ContainerStarted","Data":"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"}
Oct 30 02:54:21.179419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:21.179257    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" event={"ID":"93a18368-7423-47c1-b0dc-c2aaf3999a42","Type":"ContainerStarted","Data":"18c249b6beba35e8d303c2c545751ba5be02d6808c1fa245fea1185e2d8ff8af"}
Oct 30 02:54:21.222999 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:21.222929    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podStartSLOduration=1.222912083 podStartE2EDuration="1.222912083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:21.222268456 +0000 UTC m=+800.296439246" watchObservedRunningTime="2026-10-30 02:54:21.222912083 +0000 UTC m=+800.297082877"
Oct 30 02:54:21.681841 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:21.681810    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:29.373670 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.373634    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 02:54:29.378083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.378062    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.397936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.397911    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 02:54:29.407278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.407243    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-810119-openshift-default-dockercfg-84mrx\""
Oct 30 02:54:29.502725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502692    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.502889 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502750    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.502889 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502865    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.502966 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.502966 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502906    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.503044 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.502998    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.503044 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.503016    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75gf\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.503151 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.503129    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.503219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.503205    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604074 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604039    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604099    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604172    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604263    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604282    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604330    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604354    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75gf\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604392    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604798 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604606    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604867 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604803    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604931 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604908    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.604990 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.604929    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.605275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.605251    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.606540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.606517    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.606714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.606696    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.624767 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.624707    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.625541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.625520    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75gf\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:29.693083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:29.693043    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:30.018026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.017999    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 02:54:30.018170 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:54:30.017998    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbce61e7_9bf4_4504_b488_ca489556d1cf.slice/crio-b14b25e5f0ad441c94bad5982321013f122d920b9b1d627eb044a8f4897e9530 WatchSource:0}: Error finding container b14b25e5f0ad441c94bad5982321013f122d920b9b1d627eb044a8f4897e9530: Status 404 returned error can't find the container with id b14b25e5f0ad441c94bad5982321013f122d920b9b1d627eb044a8f4897e9530
Oct 30 02:54:30.020653 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.020621    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:30.020747 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.020688    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:30.020747 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.020719    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:30.234428 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.234378    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" event={"ID":"bbce61e7-9bf4-4504-b488-ca489556d1cf","Type":"ContainerStarted","Data":"1e1ba056b95940286ada873f2228db76cb824a4b0cefac8c65afac82b2979c8a"}
Oct 30 02:54:30.234587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.234441    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" event={"ID":"bbce61e7-9bf4-4504-b488-ca489556d1cf","Type":"ContainerStarted","Data":"b14b25e5f0ad441c94bad5982321013f122d920b9b1d627eb044a8f4897e9530"}
Oct 30 02:54:30.278556 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.278444    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podStartSLOduration=1.278426592 podStartE2EDuration="1.278426592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:30.276887815 +0000 UTC m=+809.351058610" watchObservedRunningTime="2026-10-30 02:54:30.278426592 +0000 UTC m=+809.352597386"
Oct 30 02:54:30.681689 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.681657    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:30.697411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.697323    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:30.706984 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:30.706930    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:31.281444 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:31.281403    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:54:31.695128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:31.695096    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:31.695524 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:31.695148    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:32.693929 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:32.693889    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:32.694139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:32.693951    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:33.694056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:33.694011    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:33.694467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:33.694093    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:34.693655 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:34.693621    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:34.693836 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:34.693675    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:35.693939 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:35.693901    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:35.694381 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:35.693963    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:36.059254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.057304    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 02:54:36.089499 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.089426    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 02:54:36.089980 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.089954    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.108509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.108482    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"openshift-service-ca.crt\""
Oct 30 02:54:36.121804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.121777    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"default-dockercfg-4fftw\""
Oct 30 02:54:36.121933 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.121805    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"kube-root-ca.crt\""
Oct 30 02:54:36.121933 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.121884    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"premium-w3-d9f3ac-kserve-self-signed-certs\""
Oct 30 02:54:36.205418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205380    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.205418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205425    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.205636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205452    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.205636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205528    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94v8\" (UniqueName: \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.205636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205573    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.205636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.205600    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309015    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309369    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309568 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309554    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309661    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94v8\" (UniqueName: \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309804    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.309937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.309924    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.310297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.310277    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.310494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.310357    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.310631 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.310367    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.312577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.312531    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.313053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.313031    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.342252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.339095    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94v8\" (UniqueName: \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\") pod \"premium-w3-d9f3ac-kserve-849b7f5899-mv8zh\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") " pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.407661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.407619    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:36.693721 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.693645    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 02:54:36.693859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.693711    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 02:54:36.713520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:36.713494    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 02:54:36.716055 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:54:36.716029    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbde07f9d_958a_4478_9c16_08d908902b66.slice/crio-aa4729d59c3648ae622246f7460d613d091034ab8f18d1629732e21f7786d741 WatchSource:0}: Error finding container aa4729d59c3648ae622246f7460d613d091034ab8f18d1629732e21f7786d741: Status 404 returned error can't find the container with id aa4729d59c3648ae622246f7460d613d091034ab8f18d1629732e21f7786d741
Oct 30 02:54:37.316939 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.316900    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" event={"ID":"bde07f9d-958a-4478-9c16-08d908902b66","Type":"ContainerStarted","Data":"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"}
Oct 30 02:54:37.317114 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.316958    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" event={"ID":"bde07f9d-958a-4478-9c16-08d908902b66","Type":"ContainerStarted","Data":"aa4729d59c3648ae622246f7460d613d091034ab8f18d1629732e21f7786d741"}
Oct 30 02:54:37.359314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.359261    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podStartSLOduration=1.359246392 podStartE2EDuration="1.359246392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:37.356156664 +0000 UTC m=+816.430327457" watchObservedRunningTime="2026-10-30 02:54:37.359246392 +0000 UTC m=+816.433417178"
Oct 30 02:54:37.408101 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.408068    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:37.698638 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.698567    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:37.699002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.698986    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:37.699768 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:37.699747    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 02:54:39.164737 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:54:39.164698    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:54:46.408626 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:46.408594    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:46.421478 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:46.421452    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:47.392578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:47.392546    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:54:53.129478 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.129438    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"]
Oct 30 02:54:53.148328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.144207    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.168467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.168437    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:54:53.173401 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.173380    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:54:53.180243 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.179330    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"]
Oct 30 02:54:53.233000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.232974    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.233194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.233164    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4sn4\" (UniqueName: \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.233275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.233258    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.334030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.333999    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4sn4\" (UniqueName: \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.334188 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.334043    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.334188 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.334082    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.334871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.334845    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.338186 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.338154    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.357652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.357618    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4sn4\" (UniqueName: \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\") pod \"maas-api-revoke-sub-1702e439-6v5bv\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.516413 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.516338    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:53.835294 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:53.835263    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"]
Oct 30 02:54:53.836600 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:54:53.836558    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode162a449_4ed6_4e1b_a84c_a3df776bc96a.slice/crio-d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56 WatchSource:0}: Error finding container d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56: Status 404 returned error can't find the container with id d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56
Oct 30 02:54:54.427727 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:54.427691    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv" event={"ID":"e162a449-4ed6-4e1b-a84c-a3df776bc96a","Type":"ContainerStarted","Data":"d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56"}
Oct 30 02:54:57.449639 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:57.449421    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv" event={"ID":"e162a449-4ed6-4e1b-a84c-a3df776bc96a","Type":"ContainerStarted","Data":"6a829376976d66db70cb75972ef0cc46685a19f1af08731a1a711417c423e4de"}
Oct 30 02:54:58.456216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.456184    4214 generic.go:356] "Generic (PLEG): container finished" podID="e162a449-4ed6-4e1b-a84c-a3df776bc96a" containerID="6a829376976d66db70cb75972ef0cc46685a19f1af08731a1a711417c423e4de" exitCode=0
Oct 30 02:54:58.456646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.456278    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv" event={"ID":"e162a449-4ed6-4e1b-a84c-a3df776bc96a","Type":"ContainerDied","Data":"6a829376976d66db70cb75972ef0cc46685a19f1af08731a1a711417c423e4de"}
Oct 30 02:54:58.459357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.459317    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv" podStartSLOduration=2.445841373 podStartE2EDuration="5.459305699s" totalImagesPullingTime="3.013464326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:57.498582082 +0000 UTC m=+836.572752875" watchObservedRunningTime="2026-10-30 02:54:58.459305699 +0000 UTC m=+837.533476491"
Oct 30 02:54:58.460478 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.460463    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 02:54:58.464893 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.464876    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.496157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.496138    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-32bad3-openshift-default-dockercfg-wg2rd\""
Oct 30 02:54:58.501768 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.501744    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 02:54:58.503838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503810    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.503972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503859    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.503972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.503972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503929    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.503972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503948    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.504198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.503999    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.504198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.504027    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.504198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.504085    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6hx\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.504453 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.504154    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605202    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6hx\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605278    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605375    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605409    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605461    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605491    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605563    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605641 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605608    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605848 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605785    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605904 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605852    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.605904 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.605876    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.606102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.606075    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.606309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.606291    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.607942 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.607920    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.608139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.608120    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.626833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.626808    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6hx\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.626932 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.626811    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:58.777953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:58.777876    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:59.102978 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.102951    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 02:54:59.104833 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:54:59.104806    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297a8a31_7402_4ebe_86df_bcd02ab9d096.slice/crio-68f82328b8ba07955e43bccc052f3d8f05df0191b16fadc8f8abb5edbb9acb3d WatchSource:0}: Error finding container 68f82328b8ba07955e43bccc052f3d8f05df0191b16fadc8f8abb5edbb9acb3d: Status 404 returned error can't find the container with id 68f82328b8ba07955e43bccc052f3d8f05df0191b16fadc8f8abb5edbb9acb3d
Oct 30 02:54:59.107169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.107136    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:59.107270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.107195    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:59.107270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.107224    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:54:59.465432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.465348    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" event={"ID":"297a8a31-7402-4ebe-86df-bcd02ab9d096","Type":"ContainerStarted","Data":"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"}
Oct 30 02:54:59.465432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.465404    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" event={"ID":"297a8a31-7402-4ebe-86df-bcd02ab9d096","Type":"ContainerStarted","Data":"68f82328b8ba07955e43bccc052f3d8f05df0191b16fadc8f8abb5edbb9acb3d"}
Oct 30 02:54:59.671820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.671791    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:54:59.718714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.718633    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\" (UniqueName: \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\") pod \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") "
Oct 30 02:54:59.718851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.718763    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\") pod \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") "
Oct 30 02:54:59.718851 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.718815    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\") pod \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\" (UID: \"e162a449-4ed6-4e1b-a84c-a3df776bc96a\") "
Oct 30 02:54:59.720963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.720933    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca" pod "e162a449-4ed6-4e1b-a84c-a3df776bc96a" (UID: "e162a449-4ed6-4e1b-a84c-a3df776bc96a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:54:59.722428 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.722380    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podStartSLOduration=1.722362902 podStartE2EDuration="1.722362902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:59.556549256 +0000 UTC m=+838.630720047" watchObservedRunningTime="2026-10-30 02:54:59.722362902 +0000 UTC m=+838.796533696"
Oct 30 02:54:59.723720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.723689    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4" pod "e162a449-4ed6-4e1b-a84c-a3df776bc96a" (UID: "e162a449-4ed6-4e1b-a84c-a3df776bc96a"). InnerVolumeSpecName "kube-api-access-w4sn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:54:59.724286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.724261    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca" pod "e162a449-4ed6-4e1b-a84c-a3df776bc96a" (UID: "e162a449-4ed6-4e1b-a84c-a3df776bc96a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:54:59.778410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.778377    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:54:59.822271 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.822212    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:54:59.822271 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.822267    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4sn4\" (UniqueName: \"kubernetes.io/projected/e162a449-4ed6-4e1b-a84c-a3df776bc96a-kube-api-access-w4sn4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:54:59.822497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:54:59.822284    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e162a449-4ed6-4e1b-a84c-a3df776bc96a-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:00.483307 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:00.483274    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"
Oct 30 02:55:00.483697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:00.483274    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv" event={"ID":"e162a449-4ed6-4e1b-a84c-a3df776bc96a","Type":"ContainerDied","Data":"d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56"}
Oct 30 02:55:00.483697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:00.483421    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c54619f2e9ffbaf9f5019c36299afbf132c7115fc4f14c235a7eae5dbbae56"
Oct 30 02:55:00.779693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:00.779601    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:55:00.779693 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:00.779656    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 02:55:01.778923 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:01.778879    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:55:01.779340 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:01.778949    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 02:55:02.778352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:02.778318    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:55:02.778525 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:02.778376    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 02:55:03.778839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:03.778799    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:55:03.779358 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:03.778859    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 02:55:04.778881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:04.778839    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:55:04.779410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:04.778894    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 02:55:06.660878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:06.660846    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:55:06.662627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:06.662606    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:55:06.670676 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:06.666420    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 02:55:07.599997 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.599959    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"]
Oct 30 02:55:07.600494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.600476    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e162a449-4ed6-4e1b-a84c-a3df776bc96a" containerName="revoke-keys"
Oct 30 02:55:07.600494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.600491    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e162a449-4ed6-4e1b-a84c-a3df776bc96a" containerName="revoke-keys"
Oct 30 02:55:07.600680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.600591    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e162a449-4ed6-4e1b-a84c-a3df776bc96a" containerName="revoke-keys"
Oct 30 02:55:07.604016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.604000    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.620676 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.620654    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:07.633518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.633493    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:07.637936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.637914    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"]
Oct 30 02:55:07.772487 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.772454    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.772836 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.772591    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.772836 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.772653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnp5\" (UniqueName: \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.874148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.874069    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.874148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.874114    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnp5\" (UniqueName: \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.874358 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.874215    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.874900 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.874879    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.876496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.876475    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.894694 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.894675    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnp5\" (UniqueName: \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\") pod \"maas-api-revoke-sub-57ba5cb2-lzx44\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:07.967464 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:07.967435    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:08.279599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:08.279566    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"]
Oct 30 02:55:08.279752 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:08.279721    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd046002_da02_43d8_b342_473452ebea47.slice/crio-05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d WatchSource:0}: Error finding container 05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d: Status 404 returned error can't find the container with id 05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d
Oct 30 02:55:08.537600 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:08.537551    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44" event={"ID":"bd046002-da02-43d8-b342-473452ebea47","Type":"ContainerStarted","Data":"05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d"}
Oct 30 02:55:09.166104 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:55:09.166060    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:55:09.545158 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:09.545125    4214 generic.go:356] "Generic (PLEG): container finished" podID="bd046002-da02-43d8-b342-473452ebea47" containerID="56074c74ba1fa72fcff3f6398972aa3c72460c93b47d826304331532a6fd942f" exitCode=0
Oct 30 02:55:09.545344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:09.545205    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44" event={"ID":"bd046002-da02-43d8-b342-473452ebea47","Type":"ContainerDied","Data":"56074c74ba1fa72fcff3f6398972aa3c72460c93b47d826304331532a6fd942f"}
Oct 30 02:55:10.709005 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.708979    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:10.809911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.809832    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\" (UniqueName: \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\") pod \"bd046002-da02-43d8-b342-473452ebea47\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") "
Oct 30 02:55:10.810072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.809917    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\") pod \"bd046002-da02-43d8-b342-473452ebea47\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") "
Oct 30 02:55:10.810150 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.810069    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\") pod \"bd046002-da02-43d8-b342-473452ebea47\" (UID: \"bd046002-da02-43d8-b342-473452ebea47\") "
Oct 30 02:55:10.810494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.810458    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca" pod "bd046002-da02-43d8-b342-473452ebea47" (UID: "bd046002-da02-43d8-b342-473452ebea47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:10.812056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.812004    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5" pod "bd046002-da02-43d8-b342-473452ebea47" (UID: "bd046002-da02-43d8-b342-473452ebea47"). InnerVolumeSpecName "kube-api-access-8gnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:10.812331 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.812293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca" pod "bd046002-da02-43d8-b342-473452ebea47" (UID: "bd046002-da02-43d8-b342-473452ebea47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:10.911490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.911457    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd046002-da02-43d8-b342-473452ebea47-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:10.911490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.911486    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnp5\" (UniqueName: \"kubernetes.io/projected/bd046002-da02-43d8-b342-473452ebea47-kube-api-access-8gnp5\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:10.911490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:10.911496    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd046002-da02-43d8-b342-473452ebea47-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:11.559859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:11.559835    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"
Oct 30 02:55:11.560029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:11.559883    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44" event={"ID":"bd046002-da02-43d8-b342-473452ebea47","Type":"ContainerDied","Data":"05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d"}
Oct 30 02:55:11.560029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:11.559928    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05646c3de27ddb72cf8160b142fa87b2ac0e6bcc30180059115fa6d3a137c10d"
Oct 30 02:55:20.650820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.650745    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"]
Oct 30 02:55:20.651352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.651331    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bd046002-da02-43d8-b342-473452ebea47" containerName="revoke-keys"
Oct 30 02:55:20.651433 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.651354    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd046002-da02-43d8-b342-473452ebea47" containerName="revoke-keys"
Oct 30 02:55:20.651540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.651526    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd046002-da02-43d8-b342-473452ebea47" containerName="revoke-keys"
Oct 30 02:55:20.655064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.655038    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.674132 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.674103    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"]
Oct 30 02:55:20.682713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.682688    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:20.682818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.682762    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:20.771088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.771044    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.771281 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.771118    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.771281 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.771196    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smq4l\" (UniqueName: \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.872709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.872674    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.872872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.872719    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.872872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.872740    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smq4l\" (UniqueName: \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.873538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.873504    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.875062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.875041    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.893251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.893191    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smq4l\" (UniqueName: \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\") pod \"maas-api-revoke-sub-c43ecdd1-7r4zz\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:20.967924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:20.967856    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:21.281406 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:21.281375    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439243d3_1673_4857_9edd_461f0d2c176a.slice/crio-c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405 WatchSource:0}: Error finding container c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405: Status 404 returned error can't find the container with id c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405
Oct 30 02:55:21.282714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:21.282688    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"]
Oct 30 02:55:21.630441 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:21.630391    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz" event={"ID":"439243d3-1673-4857-9edd-461f0d2c176a","Type":"ContainerStarted","Data":"6ea38a35ed2481dc2a1f41f6b533a205f92195d5ec8d60b88a69f67dc4614ef6"}
Oct 30 02:55:21.630599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:21.630464    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz" event={"ID":"439243d3-1673-4857-9edd-461f0d2c176a","Type":"ContainerStarted","Data":"c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405"}
Oct 30 02:55:21.672285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:21.672210    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz" podStartSLOduration=1.672193158 podStartE2EDuration="1.672193158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:21.669891963 +0000 UTC m=+860.744062793" watchObservedRunningTime="2026-10-30 02:55:21.672193158 +0000 UTC m=+860.746363988"
Oct 30 02:55:22.640888 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:22.640849    4214 generic.go:356] "Generic (PLEG): container finished" podID="439243d3-1673-4857-9edd-461f0d2c176a" containerID="6ea38a35ed2481dc2a1f41f6b533a205f92195d5ec8d60b88a69f67dc4614ef6" exitCode=0
Oct 30 02:55:22.641091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:22.641068    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz" event={"ID":"439243d3-1673-4857-9edd-461f0d2c176a","Type":"ContainerDied","Data":"6ea38a35ed2481dc2a1f41f6b533a205f92195d5ec8d60b88a69f67dc4614ef6"}
Oct 30 02:55:23.804119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.804088    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:23.828376 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.828347    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\") pod \"439243d3-1673-4857-9edd-461f0d2c176a\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") "
Oct 30 02:55:23.828533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.828514    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\") pod \"439243d3-1673-4857-9edd-461f0d2c176a\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") "
Oct 30 02:55:23.828599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.828564    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\" (UniqueName: \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\") pod \"439243d3-1673-4857-9edd-461f0d2c176a\" (UID: \"439243d3-1673-4857-9edd-461f0d2c176a\") "
Oct 30 02:55:23.828733 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.828712    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca" pod "439243d3-1673-4857-9edd-461f0d2c176a" (UID: "439243d3-1673-4857-9edd-461f0d2c176a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:23.828921 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.828905    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:23.830490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.830469    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca" pod "439243d3-1673-4857-9edd-461f0d2c176a" (UID: "439243d3-1673-4857-9edd-461f0d2c176a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:23.830618 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.830596    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l" pod "439243d3-1673-4857-9edd-461f0d2c176a" (UID: "439243d3-1673-4857-9edd-461f0d2c176a"). InnerVolumeSpecName "kube-api-access-smq4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:23.930369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.930296    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439243d3-1673-4857-9edd-461f0d2c176a-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:23.930369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:23.930325    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smq4l\" (UniqueName: \"kubernetes.io/projected/439243d3-1673-4857-9edd-461f0d2c176a-kube-api-access-smq4l\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:24.528673 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.528637    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"]
Oct 30 02:55:24.529157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.529145    4214 cpu_manager.go:414] "containerMap: removing container" podUID="439243d3-1673-4857-9edd-461f0d2c176a" containerName="revoke-keys"
Oct 30 02:55:24.529202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.529158    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439243d3-1673-4857-9edd-461f0d2c176a" containerName="revoke-keys"
Oct 30 02:55:24.529290 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.529281    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="439243d3-1673-4857-9edd-461f0d2c176a" containerName="revoke-keys"
Oct 30 02:55:24.532441 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.532425    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.536099 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.536064    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.536263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.536224    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.536330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.536311    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxvs\" (UniqueName: \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.554979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.554951    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"]
Oct 30 02:55:24.637526 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.637491    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxvs\" (UniqueName: \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.637714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.637605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.637714 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.637657    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.638479 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.638450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.640088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.640067    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.656064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.656042    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"
Oct 30 02:55:24.656175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.656067    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz" event={"ID":"439243d3-1673-4857-9edd-461f0d2c176a","Type":"ContainerDied","Data":"c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405"}
Oct 30 02:55:24.656175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.656119    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c40475d649e0b67c195a6967855a64689b8c501656f7319193bab1e808285405"
Oct 30 02:55:24.657561 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.657540    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxvs\" (UniqueName: \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\") pod \"maas-api-revoke-sub-49690c32-zth8c\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:24.846815 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:24.846777    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:25.149902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.149819    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"]
Oct 30 02:55:25.153459 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:25.153412    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9bbf74_0972_42c4_b806_5c3d4cb24299.slice/crio-f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f WatchSource:0}: Error finding container f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f: Status 404 returned error can't find the container with id f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f
Oct 30 02:55:25.666054 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.666016    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c" event={"ID":"1e9bbf74-0972-42c4-b806-5c3d4cb24299","Type":"ContainerStarted","Data":"f347e7af25fd1db158bd1c627b53afb3e81f0319007020381c69ab482b99f79b"}
Oct 30 02:55:25.666288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.666080    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c" event={"ID":"1e9bbf74-0972-42c4-b806-5c3d4cb24299","Type":"ContainerStarted","Data":"f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f"}
Oct 30 02:55:25.706652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.706561    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c" podStartSLOduration=1.7065464449999999 podStartE2EDuration="1.706546445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:25.705318208 +0000 UTC m=+864.779489003" watchObservedRunningTime="2026-10-30 02:55:25.706546445 +0000 UTC m=+864.780717238"
Oct 30 02:55:25.936646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.936605    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"]
Oct 30 02:55:25.941602 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.941576    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:25.953880 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.953849    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:25.954070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.953945    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2t6\" (UniqueName: \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:25.954070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.954008    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:25.959507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:25.959473    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"]
Oct 30 02:55:26.059575 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.057872    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2t6\" (UniqueName: \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.059575 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.057963    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.059575 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.058088    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.061055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.061020    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.063570 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.063538    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.081257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.081213    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2t6\" (UniqueName: \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\") pod \"maas-api-revoke-sub-7c387fc0-swsjh\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.256359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.256258    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:26.565264 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:26.562190    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faa5401_cd45_4be5_b950_1578a1cbdd4b.slice/crio-d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c WatchSource:0}: Error finding container d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c: Status 404 returned error can't find the container with id d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c
Oct 30 02:55:26.565264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.563991    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"]
Oct 30 02:55:26.675062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.675026    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh" event={"ID":"5faa5401-cd45-4be5-b950-1578a1cbdd4b","Type":"ContainerStarted","Data":"d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c"}
Oct 30 02:55:26.676746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.676720    4214 generic.go:356] "Generic (PLEG): container finished" podID="1e9bbf74-0972-42c4-b806-5c3d4cb24299" containerID="f347e7af25fd1db158bd1c627b53afb3e81f0319007020381c69ab482b99f79b" exitCode=0
Oct 30 02:55:26.676884 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:26.676767    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c" event={"ID":"1e9bbf74-0972-42c4-b806-5c3d4cb24299","Type":"ContainerDied","Data":"f347e7af25fd1db158bd1c627b53afb3e81f0319007020381c69ab482b99f79b"}
Oct 30 02:55:27.688984 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:27.687877    4214 generic.go:356] "Generic (PLEG): container finished" podID="5faa5401-cd45-4be5-b950-1578a1cbdd4b" containerID="b0c1cba1ba1332bff4b5174321c2c4e77a69a51574af4dcf7f5127f6936c2fd7" exitCode=0
Oct 30 02:55:27.688984 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:27.687960    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh" event={"ID":"5faa5401-cd45-4be5-b950-1578a1cbdd4b","Type":"ContainerDied","Data":"b0c1cba1ba1332bff4b5174321c2c4e77a69a51574af4dcf7f5127f6936c2fd7"}
Oct 30 02:55:27.909469 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:27.909445    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:28.014311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.010660    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\") pod \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") "
Oct 30 02:55:28.014311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.010840    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\") pod \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") "
Oct 30 02:55:28.014311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.010921    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\" (UniqueName: \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\") pod \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\" (UID: \"1e9bbf74-0972-42c4-b806-5c3d4cb24299\") "
Oct 30 02:55:28.016455 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.016430    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca" pod "1e9bbf74-0972-42c4-b806-5c3d4cb24299" (UID: "1e9bbf74-0972-42c4-b806-5c3d4cb24299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:28.080297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.044387    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:28.080695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.080661    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs" pod "1e9bbf74-0972-42c4-b806-5c3d4cb24299" (UID: "1e9bbf74-0972-42c4-b806-5c3d4cb24299"). InnerVolumeSpecName "kube-api-access-prxvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:28.091735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.091698    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca" pod "1e9bbf74-0972-42c4-b806-5c3d4cb24299" (UID: "1e9bbf74-0972-42c4-b806-5c3d4cb24299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:28.152441 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.152395    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bbf74-0972-42c4-b806-5c3d4cb24299-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:28.156304 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.156280    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxvs\" (UniqueName: \"kubernetes.io/projected/1e9bbf74-0972-42c4-b806-5c3d4cb24299-kube-api-access-prxvs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:28.698947 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.698906    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"
Oct 30 02:55:28.698947 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.698935    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c" event={"ID":"1e9bbf74-0972-42c4-b806-5c3d4cb24299","Type":"ContainerDied","Data":"f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f"}
Oct 30 02:55:28.699548 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.698999    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8fc6226bed0adcf17fd468d267c9e784e57187d25de89ab1f9decd6a9cee13f"
Oct 30 02:55:28.827863 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.827821    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 02:55:28.828582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.828555    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1e9bbf74-0972-42c4-b806-5c3d4cb24299" containerName="revoke-keys"
Oct 30 02:55:28.828582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.828580    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9bbf74-0972-42c4-b806-5c3d4cb24299" containerName="revoke-keys"
Oct 30 02:55:28.828807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.828779    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9bbf74-0972-42c4-b806-5c3d4cb24299" containerName="revoke-keys"
Oct 30 02:55:28.840658 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.840620    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.860261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.860218    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 02:55:28.868190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.868160    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f7d1a1-openshift-default-dockercfg-54qbh\""
Oct 30 02:55:28.879532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.879454    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:28.981743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.981654    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\" (UniqueName: \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\") pod \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") "
Oct 30 02:55:28.981903 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.981808    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\") pod \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") "
Oct 30 02:55:28.981968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.981898    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\") pod \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\" (UID: \"5faa5401-cd45-4be5-b950-1578a1cbdd4b\") "
Oct 30 02:55:28.982434 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.982042    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.982547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.982477    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca" pod "5faa5401-cd45-4be5-b950-1578a1cbdd4b" (UID: "5faa5401-cd45-4be5-b950-1578a1cbdd4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:28.982742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.982723    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.982924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.982908    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983083 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983066    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983243 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983214    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983334    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983401    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983447 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983438    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983541    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdj4\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:28.983750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.983721    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:28.985184 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.985152    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6" pod "5faa5401-cd45-4be5-b950-1578a1cbdd4b" (UID: "5faa5401-cd45-4be5-b950-1578a1cbdd4b"). InnerVolumeSpecName "kube-api-access-lq2t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:28.985337 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:28.985219    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca" pod "5faa5401-cd45-4be5-b950-1578a1cbdd4b" (UID: "5faa5401-cd45-4be5-b950-1578a1cbdd4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:29.085071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085040    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085085    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085104    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085173    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdj4\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085249    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085274    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085552 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085301    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085624 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085599    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085629    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.085735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085702    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085690    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085956    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085985    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.085900    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.086020    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faa5401-cd45-4be5-b950-1578a1cbdd4b-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:29.086111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.086043    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2t6\" (UniqueName: \"kubernetes.io/projected/5faa5401-cd45-4be5-b950-1578a1cbdd4b-kube-api-access-lq2t6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:29.087911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.087891    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.088084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.088067    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.104826 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.104802    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.105115 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.105095    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdj4\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.186780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.186742    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:29.205040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.204999    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 02:55:29.205710 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.205689    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5faa5401-cd45-4be5-b950-1578a1cbdd4b" containerName="revoke-keys"
Oct 30 02:55:29.205710 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.205710    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faa5401-cd45-4be5-b950-1578a1cbdd4b" containerName="revoke-keys"
Oct 30 02:55:29.205835 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.205807    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faa5401-cd45-4be5-b950-1578a1cbdd4b" containerName="revoke-keys"
Oct 30 02:55:29.209668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.209646    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.225729 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.225705    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e11f03-openshift-default-dockercfg-ph97z\""
Oct 30 02:55:29.227095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.227071    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 02:55:29.390972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.390940    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.390982    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391057    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wq8\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391084    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391103    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391347 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391252    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391347 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391308    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391347 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391338    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.391483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.391370    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492113 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492080    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492327 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492132    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492327 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492241    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492327 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492277    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492370    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492402    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wq8\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492439    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492459    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492612    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492669    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492763    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492905    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.492956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.492924    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.494642 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.494623    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.494846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.494830    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.531086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.530452    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.534284 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.534201    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 02:55:29.534848 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.534810    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wq8\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.542832 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:29.542782    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52d195d_da08_4ec1_a536_67d421d3f369.slice/crio-a9a4b08415ffa6d1d294116b35754cdc646f28699d64dbbc86b9563b52fbc345 WatchSource:0}: Error finding container a9a4b08415ffa6d1d294116b35754cdc646f28699d64dbbc86b9563b52fbc345: Status 404 returned error can't find the container with id a9a4b08415ffa6d1d294116b35754cdc646f28699d64dbbc86b9563b52fbc345
Oct 30 02:55:29.546644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.546098    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.546644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.546190    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.546644 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.546249    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:29.573218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.573187    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 02:55:29.587708 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.587683    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.601560 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.601533    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 02:55:29.695341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695311    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695381    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695463    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695491    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695534 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695512    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695768 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695651    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695768 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695690    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695768 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695728    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.695923 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.695803    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbns7\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.707708 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.707686    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"
Oct 30 02:55:29.707708 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.707696    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh" event={"ID":"5faa5401-cd45-4be5-b950-1578a1cbdd4b","Type":"ContainerDied","Data":"d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c"}
Oct 30 02:55:29.708189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.707751    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8192f2e85a8efa25f76cb80afda3da0e8454f00c0143a6002219f46e644a27c"
Oct 30 02:55:29.709896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.709862    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" event={"ID":"d52d195d-da08-4ec1-a536-67d421d3f369","Type":"ContainerStarted","Data":"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"}
Oct 30 02:55:29.710002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.709917    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" event={"ID":"d52d195d-da08-4ec1-a536-67d421d3f369","Type":"ContainerStarted","Data":"a9a4b08415ffa6d1d294116b35754cdc646f28699d64dbbc86b9563b52fbc345"}
Oct 30 02:55:29.758325 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.758277    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podStartSLOduration=1.7582633410000001 podStartE2EDuration="1.758263341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.754513965 +0000 UTC m=+868.828684762" watchObservedRunningTime="2026-10-30 02:55:29.758263341 +0000 UTC m=+868.832434133"
Oct 30 02:55:29.797181 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797158    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797196    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797242    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbns7\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797298    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797361    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797600 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797410    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797600 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797441    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797600 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797600 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797542    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.797797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.797677    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.798177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.798152    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.798292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.798191    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.798292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.798194    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.798292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.798256    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.800031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.800004    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.800104 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.800095    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.817921 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.817895    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbns7\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.818107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.818083    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-djd22\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:29.834992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.834967    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:29.928592 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:29.928563    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:30.156963 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:30.156931    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cf2197_3df5_43f9_b858_23e03e0db025.slice/crio-69c77f0c0fa261e4102f2c0941669d7039f841658d01d53f2e6b4f9f89c1dfed WatchSource:0}: Error finding container 69c77f0c0fa261e4102f2c0941669d7039f841658d01d53f2e6b4f9f89c1dfed: Status 404 returned error can't find the container with id 69c77f0c0fa261e4102f2c0941669d7039f841658d01d53f2e6b4f9f89c1dfed
Oct 30 02:55:30.159458 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.159425    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.159567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.159508    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.159567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.159554    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.159791 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.159766    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 02:55:30.187948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.187925    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:30.189174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.189151    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:30.189393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.189201    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:30.286146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.286102    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 02:55:30.294564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.294252    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.294564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.294339    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.294564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.294378    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:30.719192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.719106    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" event={"ID":"e79b0562-ee2d-4598-9769-f50e121aa764","Type":"ContainerStarted","Data":"91b3ea2e98bed1bdd8a28b8f566b0fec5751e9bb03ef27478b2a426bc85c45b9"}
Oct 30 02:55:30.719192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.719172    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" event={"ID":"e79b0562-ee2d-4598-9769-f50e121aa764","Type":"ContainerStarted","Data":"df42a5c1332b82b1fb70f71d809f347dc20df380832b1188d8565433d3a89227"}
Oct 30 02:55:30.720980 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.720954    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" event={"ID":"e9cf2197-3df5-43f9-b858-23e03e0db025","Type":"ContainerStarted","Data":"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"}
Oct 30 02:55:30.721101 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.721003    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" event={"ID":"e9cf2197-3df5-43f9-b858-23e03e0db025","Type":"ContainerStarted","Data":"69c77f0c0fa261e4102f2c0941669d7039f841658d01d53f2e6b4f9f89c1dfed"}
Oct 30 02:55:30.775564 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.775356    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podStartSLOduration=1.775337822 podStartE2EDuration="1.775337822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:30.773315759 +0000 UTC m=+869.847486551" watchObservedRunningTime="2026-10-30 02:55:30.775337822 +0000 UTC m=+869.849508616"
Oct 30 02:55:30.820047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.820000    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podStartSLOduration=1.819986526 podStartE2EDuration="1.819986526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:30.819070206 +0000 UTC m=+869.893241000" watchObservedRunningTime="2026-10-30 02:55:30.819986526 +0000 UTC m=+869.894157318"
Oct 30 02:55:30.870365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.870327    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:30.928778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:30.928736    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:31.192252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.191721    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:31.192252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.191783    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:31.865411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.865363    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:31.866052 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.866015    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:31.930193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.930154    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:31.930377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:31.930223    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:32.191314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.188685    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:32.191314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.188762    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:32.838626 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.838587    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:32.838892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.838864    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:32.930149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.930114    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:32.930619 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:32.930175    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:33.188305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.187665    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:33.188305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.187729    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:33.847251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.846806    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:33.847251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.846866    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:33.930027 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.929988    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:33.930350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:33.930317    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:34.191305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.190652    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:34.191305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.190716    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:34.838759 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.838721    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:34.839070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.839023    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:34.929011 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.928973    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:34.929162 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:34.929033    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:35.187736 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.187642    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:35.188335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.188303    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:35.682252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.678624    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"]
Oct 30 02:55:35.688955 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.688922    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.712253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.711511    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:35.712253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.711530    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:35.736254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.735056    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.736254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.735175    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.736254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.735203    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.862253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.850945    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.862253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.851069    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.862253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.851104    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.863485 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.863054    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:35.863485 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.863107    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:35.901493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.901446    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:35.924341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.917642    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"]
Oct 30 02:55:35.958308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.932608    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:35.958308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:35.932661    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:36.111295 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.089420    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:36.118574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.118542    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\") pod \"maas-api-revoke-sub-89089d14-jts7h\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:36.126303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.126276    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:36.190263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.188643    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:55:36.190263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.188701    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 02:55:36.595800 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.591841    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"]
Oct 30 02:55:36.839156 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.839120    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:55:36.839463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.839423    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 02:55:36.929978 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.929894    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:55:36.930215 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:36.930183    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 02:55:37.073329 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.073283    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h" event={"ID":"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3","Type":"ContainerStarted","Data":"b0941805999f4357993fb48f1746a9ee2222683237310c3b5e8876340ef3018b"}
Oct 30 02:55:37.073522 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.073350    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h" event={"ID":"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3","Type":"ContainerStarted","Data":"c066899422494cdeffdff1df25d06ac09e82597381badd97fc344c846a1b947d"}
Oct 30 02:55:37.192949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.192861    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:37.193745 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.193722    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:37.194494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.194474    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 02:55:37.247753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.247695    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h" podStartSLOduration=2.24765955 podStartE2EDuration="2.24765955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:37.122814875 +0000 UTC m=+876.196985669" watchObservedRunningTime="2026-10-30 02:55:37.24765955 +0000 UTC m=+876.321830344"
Oct 30 02:55:37.862318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.857054    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:37.862318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.858148    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:37.862318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.860501    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 02:55:37.933487 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.933460    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:37.933753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.933733    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:37.934540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:37.934519    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 02:55:38.081579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:38.081541    4214 generic.go:356] "Generic (PLEG): container finished" podID="d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" containerID="b0941805999f4357993fb48f1746a9ee2222683237310c3b5e8876340ef3018b" exitCode=0
Oct 30 02:55:38.081735 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:38.081628    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h" event={"ID":"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3","Type":"ContainerDied","Data":"b0941805999f4357993fb48f1746a9ee2222683237310c3b5e8876340ef3018b"}
Oct 30 02:55:39.177153 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:55:39.177117    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:55:39.243702 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.243673    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:39.326565 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.326531    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\") pod \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") "
Oct 30 02:55:39.326744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.326614    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\") pod \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") "
Oct 30 02:55:39.326744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.326645    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\") pod \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\" (UID: \"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3\") "
Oct 30 02:55:39.327367 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.327338    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca" pod "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" (UID: "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:39.329303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.329264    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj" pod "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" (UID: "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3"). InnerVolumeSpecName "kube-api-access-d9tmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:39.329486 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.329462    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca" pod "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" (UID: "d1ba7dc3-2eaf-475e-96a9-975b85c33fa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:39.440306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.438728    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tmj\" (UniqueName: \"kubernetes.io/projected/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-kube-api-access-d9tmj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:39.440306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.438766    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:39.440306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.438782    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:39.526365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.526324    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"]
Oct 30 02:55:39.527218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.527189    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" containerName="revoke-keys"
Oct 30 02:55:39.527218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.527215    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" containerName="revoke-keys"
Oct 30 02:55:39.527443 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.527423    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" containerName="revoke-keys"
Oct 30 02:55:39.531157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.531139    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.557704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.557675    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"]
Oct 30 02:55:39.642451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.642416    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.642623 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.642470    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrxs\" (UniqueName: \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.642623 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.642594    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.743920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.743846    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.743920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.743896    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrxs\" (UniqueName: \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.744136 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.743936    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.744499 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.744474    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.746174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.746149    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.764683 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.764660    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrxs\" (UniqueName: \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\") pod \"maas-api-revoke-sub-921a8a19-g4h84\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:39.856744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:39.856715    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:40.096703 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:40.096673    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"
Oct 30 02:55:40.096869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:40.096716    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h" event={"ID":"d1ba7dc3-2eaf-475e-96a9-975b85c33fa3","Type":"ContainerDied","Data":"c066899422494cdeffdff1df25d06ac09e82597381badd97fc344c846a1b947d"}
Oct 30 02:55:40.096869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:40.096767    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c066899422494cdeffdff1df25d06ac09e82597381badd97fc344c846a1b947d"
Oct 30 02:55:40.164715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:40.164689    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"]
Oct 30 02:55:40.170128 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:40.170100    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40afb98d_2fb4_4604_87ce_cc59f9143221.slice/crio-0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea WatchSource:0}: Error finding container 0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea: Status 404 returned error can't find the container with id 0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea
Oct 30 02:55:41.124071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:41.123971    4214 generic.go:356] "Generic (PLEG): container finished" podID="40afb98d-2fb4-4604-87ce-cc59f9143221" containerID="69db92b916efdee8bfe11a3964319010ceabf86f173808908389c4e247c803f1" exitCode=0
Oct 30 02:55:41.124537 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:41.124215    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84" event={"ID":"40afb98d-2fb4-4604-87ce-cc59f9143221","Type":"ContainerDied","Data":"69db92b916efdee8bfe11a3964319010ceabf86f173808908389c4e247c803f1"}
Oct 30 02:55:41.124537 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:41.124277    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84" event={"ID":"40afb98d-2fb4-4604-87ce-cc59f9143221","Type":"ContainerStarted","Data":"0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea"}
Oct 30 02:55:42.281118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.281088    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:42.379527 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.379494    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\" (UniqueName: \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\") pod \"40afb98d-2fb4-4604-87ce-cc59f9143221\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") "
Oct 30 02:55:42.379698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.379558    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\") pod \"40afb98d-2fb4-4604-87ce-cc59f9143221\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") "
Oct 30 02:55:42.379698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.379583    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\") pod \"40afb98d-2fb4-4604-87ce-cc59f9143221\" (UID: \"40afb98d-2fb4-4604-87ce-cc59f9143221\") "
Oct 30 02:55:42.379946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.379918    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca" pod "40afb98d-2fb4-4604-87ce-cc59f9143221" (UID: "40afb98d-2fb4-4604-87ce-cc59f9143221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:42.380264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.380212    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:42.381728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.381672    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs" pod "40afb98d-2fb4-4604-87ce-cc59f9143221" (UID: "40afb98d-2fb4-4604-87ce-cc59f9143221"). InnerVolumeSpecName "kube-api-access-2nrxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:42.382165 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.382133    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca" pod "40afb98d-2fb4-4604-87ce-cc59f9143221" (UID: "40afb98d-2fb4-4604-87ce-cc59f9143221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:42.482103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.482065    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nrxs\" (UniqueName: \"kubernetes.io/projected/40afb98d-2fb4-4604-87ce-cc59f9143221-kube-api-access-2nrxs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:42.482103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:42.482105    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40afb98d-2fb4-4604-87ce-cc59f9143221-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:43.139152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:43.139116    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84" event={"ID":"40afb98d-2fb4-4604-87ce-cc59f9143221","Type":"ContainerDied","Data":"0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea"}
Oct 30 02:55:43.139307 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:43.139162    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"
Oct 30 02:55:43.139307 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:43.139176    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0326107932fe0baf34fb03e7332caac6ca0d2d81f3d86c672ec356db17b759ea"
Oct 30 02:55:44.083504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.083466    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 02:55:44.087193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.085535    4214 cpu_manager.go:414] "containerMap: removing container" podUID="40afb98d-2fb4-4604-87ce-cc59f9143221" containerName="revoke-keys"
Oct 30 02:55:44.087193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.085562    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40afb98d-2fb4-4604-87ce-cc59f9143221" containerName="revoke-keys"
Oct 30 02:55:44.087193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.085685    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="40afb98d-2fb4-4604-87ce-cc59f9143221" containerName="revoke-keys"
Oct 30 02:55:44.124254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.108637    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.205072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.205040    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 02:55:44.292340 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.292303    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqjz\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.292863 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.292836    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.293061 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.293045    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.304054 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.304023    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.304542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.304514    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.304716 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.304701    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.304833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.304820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.304981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.304966    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.305084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.305071    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406573    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406646    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406692    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406747    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406770    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406893    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqjz\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.406924    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.407033    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.407206    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.408963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.408822    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.409883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.409629    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.409883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.409847    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.411317 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.411288    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.411537 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.411423    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.413662 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.413631    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.434134 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.434099    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.459269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.459055    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqjz\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.462958 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.461244    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:44.506915 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:44.505477    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:45.538831 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:45.517007    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 02:55:45.621445 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:45.621410    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e774eaf_0d2a_462f_ae16_ad0e86be3496.slice/crio-e21115e4e06ac410325fdae58b3e1e57a5d490f9da8dd6e01630d5c7d4d7b3f3 WatchSource:0}: Error finding container e21115e4e06ac410325fdae58b3e1e57a5d490f9da8dd6e01630d5c7d4d7b3f3: Status 404 returned error can't find the container with id e21115e4e06ac410325fdae58b3e1e57a5d490f9da8dd6e01630d5c7d4d7b3f3
Oct 30 02:55:45.625178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:45.624004    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:45.625178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:45.624092    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:45.625178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:45.624135    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:55:46.369853 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:46.369756    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" event={"ID":"5e774eaf-0d2a-462f-ae16-ad0e86be3496","Type":"ContainerStarted","Data":"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"}
Oct 30 02:55:46.370036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:46.369902    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" event={"ID":"5e774eaf-0d2a-462f-ae16-ad0e86be3496","Type":"ContainerStarted","Data":"e21115e4e06ac410325fdae58b3e1e57a5d490f9da8dd6e01630d5c7d4d7b3f3"}
Oct 30 02:55:46.431725 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:46.431662    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podStartSLOduration=2.431643207 podStartE2EDuration="2.431643207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:46.426367177 +0000 UTC m=+885.500537981" watchObservedRunningTime="2026-10-30 02:55:46.431643207 +0000 UTC m=+885.505814001"
Oct 30 02:55:46.506615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:46.506575    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:47.507834 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:47.507802    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:47.508195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:47.507853    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:48.215141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.215102    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"]
Oct 30 02:55:48.222183 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.222154    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.276318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.273799    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:55:48.276318 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.273853    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:55:48.288457 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.287364    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"]
Oct 30 02:55:48.306946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.306903    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdrr\" (UniqueName: \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.307416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.307383    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.307515 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.307486    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.408897 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.408866    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdrr\" (UniqueName: \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.409074 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.408988    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.409074 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.409023    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.409743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.409713    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.411277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.411256    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.429612 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.429582    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdrr\" (UniqueName: \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\") pod \"maas-api-revoke-sub-fb1afaee-gjvgn\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.507393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.507302    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:48.507393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.507364    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:48.589839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.589807    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:48.924653 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:48.924625    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"]
Oct 30 02:55:48.924920 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:55:48.924896    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979f96bc_700d_4802_8910_6fa66acb0a32.slice/crio-1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb WatchSource:0}: Error finding container 1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb: Status 404 returned error can't find the container with id 1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb
Oct 30 02:55:49.400855 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:49.400813    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn" event={"ID":"979f96bc-700d-4802-8910-6fa66acb0a32","Type":"ContainerStarted","Data":"1bb322577e709071a174f3b1dd22ab7a20bf14d7433b22b7a54be8b3f71fb891"}
Oct 30 02:55:49.401021 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:49.400920    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn" event={"ID":"979f96bc-700d-4802-8910-6fa66acb0a32","Type":"ContainerStarted","Data":"1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb"}
Oct 30 02:55:49.469383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:49.468853    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn" podStartSLOduration=1.4688160780000001 podStartE2EDuration="1.468816078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:49.463588723 +0000 UTC m=+888.537759518" watchObservedRunningTime="2026-10-30 02:55:49.468816078 +0000 UTC m=+888.542986872"
Oct 30 02:55:49.507914 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:49.507878    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:49.508100 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:49.507947    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:50.409536 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:50.409500    4214 generic.go:356] "Generic (PLEG): container finished" podID="979f96bc-700d-4802-8910-6fa66acb0a32" containerID="1bb322577e709071a174f3b1dd22ab7a20bf14d7433b22b7a54be8b3f71fb891" exitCode=0
Oct 30 02:55:50.409909 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:50.409554    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn" event={"ID":"979f96bc-700d-4802-8910-6fa66acb0a32","Type":"ContainerDied","Data":"1bb322577e709071a174f3b1dd22ab7a20bf14d7433b22b7a54be8b3f71fb891"}
Oct 30 02:55:50.506780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:50.506744    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:50.506945 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:50.506815    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:51.506769 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.506735    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:51.507301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.506805    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:51.613116 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.613088    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:51.770299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.769886    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\" (UniqueName: \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\") pod \"979f96bc-700d-4802-8910-6fa66acb0a32\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") "
Oct 30 02:55:51.770299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.770081    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\") pod \"979f96bc-700d-4802-8910-6fa66acb0a32\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") "
Oct 30 02:55:51.770299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.770187    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\") pod \"979f96bc-700d-4802-8910-6fa66acb0a32\" (UID: \"979f96bc-700d-4802-8910-6fa66acb0a32\") "
Oct 30 02:55:51.770881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.770851    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca" pod "979f96bc-700d-4802-8910-6fa66acb0a32" (UID: "979f96bc-700d-4802-8910-6fa66acb0a32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:51.773677 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.773643    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr" pod "979f96bc-700d-4802-8910-6fa66acb0a32" (UID: "979f96bc-700d-4802-8910-6fa66acb0a32"). InnerVolumeSpecName "kube-api-access-nqdrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:51.779446 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.779419    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca" pod "979f96bc-700d-4802-8910-6fa66acb0a32" (UID: "979f96bc-700d-4802-8910-6fa66acb0a32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:51.872076 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.872042    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:51.872076 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.872076    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979f96bc-700d-4802-8910-6fa66acb0a32-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:51.872299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:51.872088    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqdrr\" (UniqueName: \"kubernetes.io/projected/979f96bc-700d-4802-8910-6fa66acb0a32-kube-api-access-nqdrr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:55:52.426157 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:52.426126    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"
Oct 30 02:55:52.426383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:52.426120    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn" event={"ID":"979f96bc-700d-4802-8910-6fa66acb0a32","Type":"ContainerDied","Data":"1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb"}
Oct 30 02:55:52.426383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:52.426273    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1718efe442f9ac02f768a0ce88602fb79a3b288a397edda8d184f508972b1abb"
Oct 30 02:55:52.507298 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:52.507259    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:55:52.507780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:52.507322    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 02:55:53.510905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:53.510873    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:53.511396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:53.511342    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:53.512112 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:53.512090    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 02:55:58.818252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.817763    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 02:55:58.818770 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.818651    4214 cpu_manager.go:414] "containerMap: removing container" podUID="979f96bc-700d-4802-8910-6fa66acb0a32" containerName="revoke-keys"
Oct 30 02:55:58.818770 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.818670    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979f96bc-700d-4802-8910-6fa66acb0a32" containerName="revoke-keys"
Oct 30 02:55:58.818884 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.818862    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="979f96bc-700d-4802-8910-6fa66acb0a32" containerName="revoke-keys"
Oct 30 02:55:58.833805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.833588    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.975776    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.975831    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.975882    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhskm\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.975924    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.975973    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.976012    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.976091    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.976240    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:58.981253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:58.980650    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.027554 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.027518    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 02:55:59.082160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082077    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.082427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082405    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.082584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082566    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhskm\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.082712 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082695    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.082843 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.082963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.082949    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.083103 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.083087    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.083342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.083324    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.083507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.083491    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.084062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.084040    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.084847 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.084823    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.086910 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.085913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.087061 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.086139    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.087175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.086386    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.102002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.101022    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.102002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.101784    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.183839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.183803    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.204174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.204138    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhskm\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:55:59.251805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:55:59.251765    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:56:00.429061 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:00.429029    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6c48f2_cdb6_47fe_9ae9_97cdb4ae544e.slice/crio-854836d0f5043475b952e0a2c934e55a33f3f5cd00e818b0fc071d3ff8492b63 WatchSource:0}: Error finding container 854836d0f5043475b952e0a2c934e55a33f3f5cd00e818b0fc071d3ff8492b63: Status 404 returned error can't find the container with id 854836d0f5043475b952e0a2c934e55a33f3f5cd00e818b0fc071d3ff8492b63
Oct 30 02:56:00.429555 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.429530    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 02:56:00.431715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.431684    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.431838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.431782    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.431838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.431829    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:00.618354 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.618295    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" event={"ID":"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e","Type":"ContainerStarted","Data":"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"}
Oct 30 02:56:00.618547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.618364    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" event={"ID":"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e","Type":"ContainerStarted","Data":"854836d0f5043475b952e0a2c934e55a33f3f5cd00e818b0fc071d3ff8492b63"}
Oct 30 02:56:00.664967 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:00.664906    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podStartSLOduration=2.66489251 podStartE2EDuration="2.66489251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:00.660567143 +0000 UTC m=+899.734737984" watchObservedRunningTime="2026-10-30 02:56:00.66489251 +0000 UTC m=+899.739063303"
Oct 30 02:56:01.257253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:01.255288    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:56:01.257253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:01.255290    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:01.257253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:01.255446    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:02.258256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:02.254629    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:02.258256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:02.254693    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:03.254722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:03.254681    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:03.255026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:03.254983    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:04.252719 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:04.252684    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:04.253434 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:04.253399    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:05.253416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:05.253377    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:05.254026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:05.253992    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:06.256254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:06.255647    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:56:06.256254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:06.255710    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 02:56:07.260451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:07.260416    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:56:07.261308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:07.261275    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:56:07.262356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:07.262331    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 02:56:09.165779 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:56:09.165739    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:56:13.924687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.924651    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 02:56:13.930848 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.930822    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:13.952675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.952649    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"openshift-service-ca.crt\""
Oct 30 02:56:13.953996 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.953971    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 02:56:13.960904 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.960883    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"kube-root-ca.crt\""
Oct 30 02:56:13.961053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.960948    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"default-dockercfg-98cf4\""
Oct 30 02:56:13.965998 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.965976    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"distinct-w2-810119-kserve-self-signed-certs\""
Oct 30 02:56:13.977943 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.977920    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 02:56:13.982765 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.982742    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:13.993339 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:13.993294    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 02:56:14.066326 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066289    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066334    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066429    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066457    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066493    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066512    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066557    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066580    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066607    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066634    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4kk\" (UniqueName: \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066648    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066674    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.066731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066696    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jfv\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.067148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066752    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.067148 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.066779    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168004 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.167976    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168013    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168043    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168080    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168095    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168192 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168505 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168297    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168505 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168427    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168505 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168473    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168505 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168487    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168522    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168569    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4kk\" (UniqueName: \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168594    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168638    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168696 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168672    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jfv\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168938 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168764    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168938 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168787    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.168938 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168871    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.168938 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.168924    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.169139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.169115    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.169199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.169151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.169482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.169431    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.169578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.169497    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.170970 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.170948    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.171066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.171054    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.171466 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.171444    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.171560 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.171505    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.188924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.188868    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jfv\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.189060 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.189041    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.189293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.189273    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4kk\" (UniqueName: \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\") pod \"distinct-w2-810119-kserve-68bf5b7577-xwfcx\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") " pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.278391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.276297    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:14.311260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.310369    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:14.622941 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.622910    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 02:56:14.627681 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:14.627655    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04c0f412_50d8_4e14_9c47_4275c9def39e.slice/crio-f4bde61434a0a2c958c83f8fdd944f185e00df12f3b2092c6dee780f8413280d WatchSource:0}: Error finding container f4bde61434a0a2c958c83f8fdd944f185e00df12f3b2092c6dee780f8413280d: Status 404 returned error can't find the container with id f4bde61434a0a2c958c83f8fdd944f185e00df12f3b2092c6dee780f8413280d
Oct 30 02:56:14.649122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.649091    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 02:56:14.652191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.652159    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:14.652338 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.652316    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:14.652384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.652371    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:14.722753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.722726    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 02:56:14.728778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.728750    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.767418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.767381    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" event={"ID":"04c0f412-50d8-4e14-9c47-4275c9def39e","Type":"ContainerStarted","Data":"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"}
Oct 30 02:56:14.767580 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.767433    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" event={"ID":"04c0f412-50d8-4e14-9c47-4275c9def39e","Type":"ContainerStarted","Data":"f4bde61434a0a2c958c83f8fdd944f185e00df12f3b2092c6dee780f8413280d"}
Oct 30 02:56:14.773410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.773370    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" event={"ID":"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0","Type":"ContainerStarted","Data":"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"}
Oct 30 02:56:14.773615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.773599    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" event={"ID":"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0","Type":"ContainerStarted","Data":"57d0a83d9513d6582310849edcd0fd3a70e1d00ee95f096d1cf1d622d2641884"}
Oct 30 02:56:14.774701 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.774677    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 02:56:14.791311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.791271    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.791609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.791590    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.791810 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.791795    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.791977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.791961    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.792138 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.792123    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.792334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.792318    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.792515 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.792501    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.792672 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.792657    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdf7q\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.792826 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.792810    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.829993 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.829923    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podStartSLOduration=1.8299034600000001 podStartE2EDuration="1.82990346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:14.825507644 +0000 UTC m=+913.899678428" watchObservedRunningTime="2026-10-30 02:56:14.82990346 +0000 UTC m=+913.904074255"
Oct 30 02:56:14.870719 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.870661    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podStartSLOduration=1.870646523 podStartE2EDuration="1.870646523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:14.867803302 +0000 UTC m=+913.941974095" watchObservedRunningTime="2026-10-30 02:56:14.870646523 +0000 UTC m=+913.944817317"
Oct 30 02:56:14.894691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.894600    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.894691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.894658    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.894912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.894710    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895027    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895079    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895111    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895189    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895245    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895275    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdf7q\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895304    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895720    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896102 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.895952    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896694 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.896197    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.896757 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.896736    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.897682 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.897653    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.898486 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.898461    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.916189 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.916164    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:14.916777 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:14.916751    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdf7q\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:15.045254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.045211    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:15.276994 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.276910    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:15.311278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.311210    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:15.312902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.312870    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:15.313033 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.312929    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:15.369294 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.363218    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 02:56:15.373579 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:15.373537    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee1419b_2461_4811_8e23_2fd1905bdb0e.slice/crio-770d55591ff4237d55f018e0297620f8367d8e5b3babe6b3f14541bd564b1fc8 WatchSource:0}: Error finding container 770d55591ff4237d55f018e0297620f8367d8e5b3babe6b3f14541bd564b1fc8: Status 404 returned error can't find the container with id 770d55591ff4237d55f018e0297620f8367d8e5b3babe6b3f14541bd564b1fc8
Oct 30 02:56:15.376680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.376636    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.376755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.376738    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.376799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.376781    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:15.788350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.788311    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" event={"ID":"aee1419b-2461-4811-8e23-2fd1905bdb0e","Type":"ContainerStarted","Data":"04567e9b203076f5156733e697b6f0ea950aadeea120d92a53f8f347f1eb623c"}
Oct 30 02:56:15.788684 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.788661    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" event={"ID":"aee1419b-2461-4811-8e23-2fd1905bdb0e","Type":"ContainerStarted","Data":"770d55591ff4237d55f018e0297620f8367d8e5b3babe6b3f14541bd564b1fc8"}
Oct 30 02:56:15.832688 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:15.832640    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podStartSLOduration=1.8326263520000001 podStartE2EDuration="1.832626352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:15.830684835 +0000 UTC m=+914.904855628" watchObservedRunningTime="2026-10-30 02:56:15.832626352 +0000 UTC m=+914.906797144"
Oct 30 02:56:16.053485 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:16.053401    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:16.054081 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:16.054054    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:16.054259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:16.054214    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:16.311590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:16.311512    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:16.311590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:16.311560    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:17.060618 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:17.060580    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:17.061209 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:17.061176    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:17.311457 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:17.311376    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:17.311457 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:17.311426    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:18.049774 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:18.049737    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:18.050063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:18.050022    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:18.311892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:18.311810    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:18.311892 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:18.311859    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:19.047254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:19.045597    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:19.047254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:19.045673    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:19.311249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:19.311145    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:19.311249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:19.311193    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:20.048043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:20.048007    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:20.048680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:20.048650    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:20.311164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:20.311080    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:20.311164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:20.311143    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:21.051254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:21.045565    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:21.051254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:21.045624    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:21.311039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:21.310924    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 02:56:21.311039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:21.310994    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 02:56:22.046173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:22.046133    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:56:22.046393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:22.046203    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 02:56:22.315680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:22.315600    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:22.316089 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:22.316035    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:22.316697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:22.316678    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 02:56:23.051961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:23.051926    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:23.052431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:23.052407    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:23.053169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:23.053152    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 02:56:24.277177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:24.277143    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:24.290313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:24.290286    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:24.931543 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:24.931515    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:56:27.391353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:27.391309    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:56:27.416541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:27.416485    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:29.635245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.635196    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 02:56:29.659498 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.646386    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.698856 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.698807    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.698966    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699031    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699060    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699083    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699105    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699133    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699194    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.699261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.699254    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.707609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.707572    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 02:56:29.800444 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800401    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800444 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800444    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800499    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800585    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800623    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800650    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800676    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800703    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.800755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.800727    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.801096 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.801009    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.801181 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.801151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.801323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.801219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.801396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.801378    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.801616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.801594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.803260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.803212    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.803367 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.803260    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.821075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.821049    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.821220 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.821201    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:29.989163 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:29.989067    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:30.312064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:30.312035    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 02:56:30.312459 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:30.312435    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda558758_44e3_4838_ae46_7a5c1da799ab.slice/crio-ae35acc79610409013cca2e39a95c1ca43c0a49b6d76f514b80111ddd6852467 WatchSource:0}: Error finding container ae35acc79610409013cca2e39a95c1ca43c0a49b6d76f514b80111ddd6852467: Status 404 returned error can't find the container with id ae35acc79610409013cca2e39a95c1ca43c0a49b6d76f514b80111ddd6852467
Oct 30 02:56:30.314784 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:30.314749    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:30.314902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:30.314830    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:30.314902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:30.314870    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:31.083073 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.083025    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" event={"ID":"da558758-44e3-4838-ae46-7a5c1da799ab","Type":"ContainerStarted","Data":"a339793b6d54cc6d352473a811285a1bbe0eefce70a0965086b54e3d764983b1"}
Oct 30 02:56:31.083535 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.083514    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" event={"ID":"da558758-44e3-4838-ae46-7a5c1da799ab","Type":"ContainerStarted","Data":"ae35acc79610409013cca2e39a95c1ca43c0a49b6d76f514b80111ddd6852467"}
Oct 30 02:56:31.150255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.148474    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podStartSLOduration=2.14845217 podStartE2EDuration="2.14845217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:31.133924511 +0000 UTC m=+930.208095314" watchObservedRunningTime="2026-10-30 02:56:31.14845217 +0000 UTC m=+930.222622967"
Oct 30 02:56:31.989282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.989214    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:31.992000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.991966    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 02:56:31.992155 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:31.992025    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 02:56:32.992258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:32.989595    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 02:56:32.992258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:32.989659    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 02:56:33.991248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:33.990277    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 02:56:33.991248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:33.990334    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 02:56:34.989680 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:34.989639    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 02:56:34.990090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:34.989699    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 02:56:35.990386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:35.990341    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 02:56:35.990924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:35.990410    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 02:56:37.013813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:37.013764    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:37.015287 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:37.015141    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:37.024948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:37.024912    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 02:56:39.186543 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:56:39.186499    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:56:43.925496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.925414    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 02:56:43.931813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.931787    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.976172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.976133    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 02:56:43.981510 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981475    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981524    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981684    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxvc\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981728    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981764    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981794    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.981842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981830    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.982160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981851    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:43.982160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:43.981890    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.082773 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082734    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.082773 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082774    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082838    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxvc\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082865    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082897    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082928    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082957    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083047 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.082980    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083045    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083209    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083223    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083622    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083889    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.083981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.083913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.085432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.085405    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.085756 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.085735    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.102497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.102476    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.102648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.102630    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxvc\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.245513 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.245423    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 02:56:44.251199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.251170    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.273344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.273315    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:44.276040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.276009    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 02:56:44.295669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.295638    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh744\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.295991 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.295966    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.296328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.296306    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.296630 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.296609    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.296771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.296754    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.296885 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.296868    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.297216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.297196    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.297427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.297408    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.297591 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.297573    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399147 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399107    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399179    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh744\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399266    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399305    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399349    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399380    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399405    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399511    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399579    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399838    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.399972 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399868    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.400034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.399964    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.400262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.400191    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.400394 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.400375    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.402483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.402458    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.402902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.402677    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.418544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.418518    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.418854 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.418837    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh744\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.575814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.575780    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:44.621764 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:44.621732    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb72f1b6_3856_4efe_8ad1_e22d0cf22b60.slice/crio-d10b530b1e265cf71b7c25344bf31aa0a5614846248ad39db153d1c46a2f2f0f WatchSource:0}: Error finding container d10b530b1e265cf71b7c25344bf31aa0a5614846248ad39db153d1c46a2f2f0f: Status 404 returned error can't find the container with id d10b530b1e265cf71b7c25344bf31aa0a5614846248ad39db153d1c46a2f2f0f
Oct 30 02:56:44.622754 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.622464    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 02:56:44.624624 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.624594    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.624717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.624661    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.624717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.624693    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.906825 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.906795    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 02:56:44.907121 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:44.907097    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d30799_48f2_4909_841b_cb2f91fffd0e.slice/crio-237cacbb900f06691ad028d14844964ddeb19e9773f1b604abd26ef636466c5e WatchSource:0}: Error finding container 237cacbb900f06691ad028d14844964ddeb19e9773f1b604abd26ef636466c5e: Status 404 returned error can't find the container with id 237cacbb900f06691ad028d14844964ddeb19e9773f1b604abd26ef636466c5e
Oct 30 02:56:44.909985 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.909945    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.910072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.910015    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:44.910111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:44.910095    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:56:45.226024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.225921    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" event={"ID":"09d30799-48f2-4909-841b-cb2f91fffd0e","Type":"ContainerStarted","Data":"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"}
Oct 30 02:56:45.226024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.225986    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" event={"ID":"09d30799-48f2-4909-841b-cb2f91fffd0e","Type":"ContainerStarted","Data":"237cacbb900f06691ad028d14844964ddeb19e9773f1b604abd26ef636466c5e"}
Oct 30 02:56:45.227948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.227917    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" event={"ID":"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60","Type":"ContainerStarted","Data":"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"}
Oct 30 02:56:45.228080 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.227964    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" event={"ID":"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60","Type":"ContainerStarted","Data":"d10b530b1e265cf71b7c25344bf31aa0a5614846248ad39db153d1c46a2f2f0f"}
Oct 30 02:56:45.277251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.276380    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:45.277251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.276791    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podStartSLOduration=1.2767712310000001 podStartE2EDuration="1.276771231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.271739608 +0000 UTC m=+944.345910403" watchObservedRunningTime="2026-10-30 02:56:45.276771231 +0000 UTC m=+944.350942025"
Oct 30 02:56:45.279303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.278114    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:45.279303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.278170    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:45.319424 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.319372    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podStartSLOduration=2.319358387 podStartE2EDuration="2.319358387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.313805286 +0000 UTC m=+944.387976086" watchObservedRunningTime="2026-10-30 02:56:45.319358387 +0000 UTC m=+944.393529181"
Oct 30 02:56:45.586058 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.585294    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:45.600289 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.594347    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:45.600289 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:45.594407    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:46.279253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:46.278601    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.279253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:46.278662    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:46.577254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:46.576637    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.577254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:46.576699    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:47.277253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:47.276559    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.277253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:47.276614    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:47.580267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:47.579663    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.580267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:47.579744    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:48.280124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:48.280086    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.280450 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:48.280415    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:48.596694 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:48.596658    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.610269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:48.597275    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:49.312262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:49.308416    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.312262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:49.308488    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:49.590919 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:49.590822    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.590919 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:49.590901    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:50.274973 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:50.274679    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.274973 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:50.274749    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:50.585260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:50.585155    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.585260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:50.585211    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:51.274638 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:51.274602    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.274931 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:51.274904    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 02:56:51.595253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:51.589921    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.595253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:51.589977    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 02:56:52.357697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.357661    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:52.358342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.358319    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:52.359717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.359693    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 02:56:52.587975 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.587949    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:52.588466 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.588437    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:52.589191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:52.589172    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 02:56:56.630315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.630279    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 02:56:56.725797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.725770    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:56.783260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.770298    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"distinct-2-w2-810119-kserve-self-signed-certs\""
Oct 30 02:56:56.783584 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.778652    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 02:56:56.932331 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.924089    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:56.959253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:56.933695    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.073250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.071801    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.073250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.071844    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.073250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.071909    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.073250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.071932    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.177742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.177704    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.178017 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.177996    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.197385 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.197302    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.198021 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.197995    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.198948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.198927    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.199609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.199586    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.200328 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.198868    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.204259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.200837    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.204633 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.198553    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.204748 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.204586    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.222684 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.222658    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.282251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.258931    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\") pod \"distinct-2-w2-810119-kserve-6777748b54-cjq8x\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") " pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:57.407505 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:57.407462    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:58.010373 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:58.010340    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 02:56:58.012940 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:58.012901    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44790c98_f4de_4749_bc91_6170be8351fc.slice/crio-9d3551b66af3f5638a55532a5bb78519ab32a65d7a3648f932c654e0b8bec2b3 WatchSource:0}: Error finding container 9d3551b66af3f5638a55532a5bb78519ab32a65d7a3648f932c654e0b8bec2b3: Status 404 returned error can't find the container with id 9d3551b66af3f5638a55532a5bb78519ab32a65d7a3648f932c654e0b8bec2b3
Oct 30 02:56:58.635123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:58.635074    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" event={"ID":"44790c98-f4de-4749-bc91-6170be8351fc","Type":"ContainerStarted","Data":"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"}
Oct 30 02:56:58.635123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:58.635126    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" event={"ID":"44790c98-f4de-4749-bc91-6170be8351fc","Type":"ContainerStarted","Data":"9d3551b66af3f5638a55532a5bb78519ab32a65d7a3648f932c654e0b8bec2b3"}
Oct 30 02:56:58.682015 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:58.681961    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podStartSLOduration=2.681947941 podStartE2EDuration="2.681947941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:58.678935307 +0000 UTC m=+957.753106099" watchObservedRunningTime="2026-10-30 02:56:58.681947941 +0000 UTC m=+957.756118732"
Oct 30 02:56:59.287110 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.287078    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"]
Oct 30 02:56:59.291286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.291267    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.313119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.312102    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"]
Oct 30 02:56:59.317440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.317403    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:56:59.317550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.317449    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:56:59.396911 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.396877    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.397101 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.396991    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxnd\" (UniqueName: \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.397240 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.397207    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.409722 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.409697    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:56:59.498769 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.498738    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxnd\" (UniqueName: \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.498954 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.498875    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.498954 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.498935    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.499713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.499691    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.501353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.501334    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.573024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.572986    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxnd\" (UniqueName: \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\") pod \"maas-api-revoke-sub-5326539f-6v6df\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.609934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.609895    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:56:59.926709 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:56:59.926677    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"]
Oct 30 02:56:59.927567 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:56:59.927536    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acdf987_90a7_4d9e_98f6_9895ff22ddd2.slice/crio-36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53 WatchSource:0}: Error finding container 36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53: Status 404 returned error can't find the container with id 36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53
Oct 30 02:57:00.655959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:00.655866    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df" event={"ID":"1acdf987-90a7-4d9e-98f6-9895ff22ddd2","Type":"ContainerStarted","Data":"9a466676062273900d42ec2a1ee645d0e4712b8224ae899818a76c47c19c65a5"}
Oct 30 02:57:00.655959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:00.655931    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df" event={"ID":"1acdf987-90a7-4d9e-98f6-9895ff22ddd2","Type":"ContainerStarted","Data":"36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53"}
Oct 30 02:57:00.699454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:00.699396    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df" podStartSLOduration=1.699382435 podStartE2EDuration="1.699382435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:00.697774023 +0000 UTC m=+959.771944814" watchObservedRunningTime="2026-10-30 02:57:00.699382435 +0000 UTC m=+959.773553227"
Oct 30 02:57:01.666184 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:01.666149    4214 generic.go:356] "Generic (PLEG): container finished" podID="1acdf987-90a7-4d9e-98f6-9895ff22ddd2" containerID="9a466676062273900d42ec2a1ee645d0e4712b8224ae899818a76c47c19c65a5" exitCode=0
Oct 30 02:57:01.666675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:01.666256    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df" event={"ID":"1acdf987-90a7-4d9e-98f6-9895ff22ddd2","Type":"ContainerDied","Data":"9a466676062273900d42ec2a1ee645d0e4712b8224ae899818a76c47c19c65a5"}
Oct 30 02:57:02.912071 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.912046    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:57:02.970297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.970203    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\" (UniqueName: \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\") pod \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") "
Oct 30 02:57:02.970454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.970412    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\") pod \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") "
Oct 30 02:57:02.970645 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.970623    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\") pod \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\" (UID: \"1acdf987-90a7-4d9e-98f6-9895ff22ddd2\") "
Oct 30 02:57:02.971509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.971479    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca" pod "1acdf987-90a7-4d9e-98f6-9895ff22ddd2" (UID: "1acdf987-90a7-4d9e-98f6-9895ff22ddd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:02.974106 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.974076    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd" pod "1acdf987-90a7-4d9e-98f6-9895ff22ddd2" (UID: "1acdf987-90a7-4d9e-98f6-9895ff22ddd2"). InnerVolumeSpecName "kube-api-access-qkxnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:02.976468 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:02.976439    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca" pod "1acdf987-90a7-4d9e-98f6-9895ff22ddd2" (UID: "1acdf987-90a7-4d9e-98f6-9895ff22ddd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:03.079244 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.079204    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:03.079510 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.079266    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:03.079510 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.079283    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxnd\" (UniqueName: \"kubernetes.io/projected/1acdf987-90a7-4d9e-98f6-9895ff22ddd2-kube-api-access-qkxnd\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:03.700186 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.700157    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"
Oct 30 02:57:03.700186 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.700180    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df" event={"ID":"1acdf987-90a7-4d9e-98f6-9895ff22ddd2","Type":"ContainerDied","Data":"36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53"}
Oct 30 02:57:03.700436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:03.700223    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a25d29bc3368a8d93b19967f157958fab938f9094503d0a30812442d4b2d53"
Oct 30 02:57:07.419891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:07.419850    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:07.439269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:07.439224    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:08.235533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:08.235503    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:09.414254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:09.411144    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.434262 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:57:09.427418    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:57:10.065266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:10.063525    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.287477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:10.287435    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.419514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:10.419413    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.444265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:10.429008    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:11.111387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.108678    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:57:11.194433 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.194400    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:11.194698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.194683    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:11.345250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.326152    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:57:11.345250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.329798    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:11.348698 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.348671    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:11.494310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:11.491894    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:13.457871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.457838    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"]
Oct 30 02:57:13.458547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.458530    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1acdf987-90a7-4d9e-98f6-9895ff22ddd2" containerName="revoke-keys"
Oct 30 02:57:13.458547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.458547    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acdf987-90a7-4d9e-98f6-9895ff22ddd2" containerName="revoke-keys"
Oct 30 02:57:13.458695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.458684    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acdf987-90a7-4d9e-98f6-9895ff22ddd2" containerName="revoke-keys"
Oct 30 02:57:13.463483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.463464    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.482104 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.482072    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"]
Oct 30 02:57:13.489794 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.489772    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:13.489950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.489780    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:13.520531 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.520501    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjht\" (UniqueName: \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.520717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.520679    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.520782 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.520738    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.622427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.622392    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.622616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.622444    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.622616 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.622584    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjht\" (UniqueName: \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.623202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.623175    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.624807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.624783    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.642044 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.642019    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjht\" (UniqueName: \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\") pod \"maas-api-revoke-sub-94cc9662-fsr4w\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:13.781891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:13.781818    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:14.092724 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:14.092690    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d3cc44_5c64_40fe_9028_05e864b938f3.slice/crio-2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657 WatchSource:0}: Error finding container 2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657: Status 404 returned error can't find the container with id 2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657
Oct 30 02:57:14.093024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:14.092999    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"]
Oct 30 02:57:14.613862 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:14.612903    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w" event={"ID":"24d3cc44-5c64-40fe-9028-05e864b938f3","Type":"ContainerStarted","Data":"ab4c9a61f2c56ab8b7262c2ad75e65acd2abd8a1e83c914468bbc6a64358c820"}
Oct 30 02:57:14.613862 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:14.612966    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w" event={"ID":"24d3cc44-5c64-40fe-9028-05e864b938f3","Type":"ContainerStarted","Data":"2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657"}
Oct 30 02:57:15.620978 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:15.620944    4214 generic.go:356] "Generic (PLEG): container finished" podID="24d3cc44-5c64-40fe-9028-05e864b938f3" containerID="ab4c9a61f2c56ab8b7262c2ad75e65acd2abd8a1e83c914468bbc6a64358c820" exitCode=0
Oct 30 02:57:15.621404 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:15.620982    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w" event={"ID":"24d3cc44-5c64-40fe-9028-05e864b938f3","Type":"ContainerDied","Data":"ab4c9a61f2c56ab8b7262c2ad75e65acd2abd8a1e83c914468bbc6a64358c820"}
Oct 30 02:57:16.806608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.806574    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:16.869916 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.869882    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\") pod \"24d3cc44-5c64-40fe-9028-05e864b938f3\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") "
Oct 30 02:57:16.870094 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.869981    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\" (UniqueName: \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\") pod \"24d3cc44-5c64-40fe-9028-05e864b938f3\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") "
Oct 30 02:57:16.870205 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.870190    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\") pod \"24d3cc44-5c64-40fe-9028-05e864b938f3\" (UID: \"24d3cc44-5c64-40fe-9028-05e864b938f3\") "
Oct 30 02:57:16.871297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.871268    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca" pod "24d3cc44-5c64-40fe-9028-05e864b938f3" (UID: "24d3cc44-5c64-40fe-9028-05e864b938f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:16.872724 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.872702    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca" pod "24d3cc44-5c64-40fe-9028-05e864b938f3" (UID: "24d3cc44-5c64-40fe-9028-05e864b938f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:16.877379 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.877357    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht" pod "24d3cc44-5c64-40fe-9028-05e864b938f3" (UID: "24d3cc44-5c64-40fe-9028-05e864b938f3"). InnerVolumeSpecName "kube-api-access-6zjht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:16.971948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.971909    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:16.971948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.971939    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d3cc44-5c64-40fe-9028-05e864b938f3-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:16.971948 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:16.971949    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjht\" (UniqueName: \"kubernetes.io/projected/24d3cc44-5c64-40fe-9028-05e864b938f3-kube-api-access-6zjht\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:17.636949 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:17.636919    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"
Oct 30 02:57:17.637129 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:17.636950    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w" event={"ID":"24d3cc44-5c64-40fe-9028-05e864b938f3","Type":"ContainerDied","Data":"2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657"}
Oct 30 02:57:17.637129 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:17.637017    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b060b0db4ad976c0d3415dd135c80cea9c7e888f6e6aeb98d8e8ce879efd657"
Oct 30 02:57:22.145658 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.145624    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"]
Oct 30 02:57:22.146263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.146223    4214 cpu_manager.go:414] "containerMap: removing container" podUID="24d3cc44-5c64-40fe-9028-05e864b938f3" containerName="revoke-keys"
Oct 30 02:57:22.146263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.146252    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d3cc44-5c64-40fe-9028-05e864b938f3" containerName="revoke-keys"
Oct 30 02:57:22.146418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.146358    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d3cc44-5c64-40fe-9028-05e864b938f3" containerName="revoke-keys"
Oct 30 02:57:22.149830 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.149809    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.167288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.167265    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:22.167816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.167795    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:22.170918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.170896    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"]
Oct 30 02:57:22.253475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.253440    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkznm\" (UniqueName: \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.253656 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.253486    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.253704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.253653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.355245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.355193    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.355449 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.355383    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.355577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.355560    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkznm\" (UniqueName: \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.355900 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.355868    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.358354 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.358323    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.376198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.376172    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkznm\" (UniqueName: \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\") pod \"maas-api-revoke-sub-2a1f5433-8dzrq\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.468181 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.468096    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:22.773612 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:22.773578    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719f90bf_9cb3_4c6d_b802_c68c32b3c4e6.slice/crio-3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617 WatchSource:0}: Error finding container 3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617: Status 404 returned error can't find the container with id 3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617
Oct 30 02:57:22.774846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:22.774814    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"]
Oct 30 02:57:23.735390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:23.735351    4214 generic.go:356] "Generic (PLEG): container finished" podID="719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" containerID="43d06bf77ff6a1dfb51828fdf2b7d504a9e373dcf9fea1df9482c7cc5148cc65" exitCode=0
Oct 30 02:57:23.735737 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:23.735441    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq" event={"ID":"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6","Type":"ContainerDied","Data":"43d06bf77ff6a1dfb51828fdf2b7d504a9e373dcf9fea1df9482c7cc5148cc65"}
Oct 30 02:57:23.735737 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:23.735499    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq" event={"ID":"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6","Type":"ContainerStarted","Data":"3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617"}
Oct 30 02:57:24.430053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.430021    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"]
Oct 30 02:57:24.434324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.434302    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.455132 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.455104    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"]
Oct 30 02:57:24.493772 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.493735    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkm8\" (UniqueName: \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.493920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.493783    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.494007 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.493990    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.595121 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.595088    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkm8\" (UniqueName: \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.595356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.595131    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.595356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.595320    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.595968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.595946    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.597540 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.597517    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.614542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.614520    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkm8\" (UniqueName: \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\") pod \"maas-api-revoke-sub-c6954229-xtmk4\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:24.752541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.752506    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:25.002740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:24.987815    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:25.049278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.049245    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\") pod \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") "
Oct 30 02:57:25.049473 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.049355    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\" (UniqueName: \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\") pod \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") "
Oct 30 02:57:25.049473 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.049459    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\") pod \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\" (UID: \"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6\") "
Oct 30 02:57:25.050255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.050212    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca" pod "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" (UID: "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:25.050459 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.050440    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:25.052186 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.052153    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca" pod "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" (UID: "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:25.052330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.052307    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm" pod "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" (UID: "719f90bf-9cb3-4c6d-b802-c68c32b3c4e6"). InnerVolumeSpecName "kube-api-access-qkznm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:25.093882 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.093856    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"]
Oct 30 02:57:25.152135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.152102    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:25.152251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.152139    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkznm\" (UniqueName: \"kubernetes.io/projected/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6-kube-api-access-qkznm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:25.754771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.754725    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4" event={"ID":"0b9dc441-8bc0-421b-866f-f509478a57dd","Type":"ContainerStarted","Data":"0f38a417a4cba557a736611a1c147b807d262d5aff5b0e247638e0eb2ad004c8"}
Oct 30 02:57:25.755286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.754828    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4" event={"ID":"0b9dc441-8bc0-421b-866f-f509478a57dd","Type":"ContainerStarted","Data":"733c3d22ccda8f4975121406e49faf6cd272393ce2b6e4e2011949cbe24fbd7d"}
Oct 30 02:57:25.756394 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.756367    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"
Oct 30 02:57:25.756514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.756365    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq" event={"ID":"719f90bf-9cb3-4c6d-b802-c68c32b3c4e6","Type":"ContainerDied","Data":"3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617"}
Oct 30 02:57:25.756514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.756495    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4a15b2406727eede9112bf4c1f57062e79e7634617592b7d0c93f899770617"
Oct 30 02:57:25.795064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:25.795005    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4" podStartSLOduration=1.794990298 podStartE2EDuration="1.794990298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:25.794029258 +0000 UTC m=+984.868200052" watchObservedRunningTime="2026-10-30 02:57:25.794990298 +0000 UTC m=+984.869161068"
Oct 30 02:57:26.764899 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:26.764860    4214 generic.go:356] "Generic (PLEG): container finished" podID="0b9dc441-8bc0-421b-866f-f509478a57dd" containerID="0f38a417a4cba557a736611a1c147b807d262d5aff5b0e247638e0eb2ad004c8" exitCode=0
Oct 30 02:57:26.765441 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:26.764907    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4" event={"ID":"0b9dc441-8bc0-421b-866f-f509478a57dd","Type":"ContainerDied","Data":"0f38a417a4cba557a736611a1c147b807d262d5aff5b0e247638e0eb2ad004c8"}
Oct 30 02:57:27.271898 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.271861    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"]
Oct 30 02:57:27.272587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.272567    4214 cpu_manager.go:414] "containerMap: removing container" podUID="719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" containerName="revoke-keys"
Oct 30 02:57:27.272587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.272588    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" containerName="revoke-keys"
Oct 30 02:57:27.272760 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.272731    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" containerName="revoke-keys"
Oct 30 02:57:27.276694 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.276672    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.295313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.295289    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"]
Oct 30 02:57:27.382979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.382893    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7rg\" (UniqueName: \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.382979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.382958    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.383174 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.383111    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.484779 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.484745    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7rg\" (UniqueName: \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.485059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.484820    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.485059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.484901    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.485859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.485827    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.487558 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.487529    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.510282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.505169    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7rg\" (UniqueName: \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\") pod \"maas-api-revoke-sub-ab1b35eb-t48wn\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.592055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.592007    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:27.906408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.906375    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"]
Oct 30 02:57:27.986136 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:27.986111    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:28.093998 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.093969    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\" (UniqueName: \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\") pod \"0b9dc441-8bc0-421b-866f-f509478a57dd\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") "
Oct 30 02:57:28.094179 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.094062    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\") pod \"0b9dc441-8bc0-421b-866f-f509478a57dd\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") "
Oct 30 02:57:28.094344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.094256    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\") pod \"0b9dc441-8bc0-421b-866f-f509478a57dd\" (UID: \"0b9dc441-8bc0-421b-866f-f509478a57dd\") "
Oct 30 02:57:28.094517 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.094480    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca" pod "0b9dc441-8bc0-421b-866f-f509478a57dd" (UID: "0b9dc441-8bc0-421b-866f-f509478a57dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:28.094887 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.094866    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:28.096178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.096154    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8" pod "0b9dc441-8bc0-421b-866f-f509478a57dd" (UID: "0b9dc441-8bc0-421b-866f-f509478a57dd"). InnerVolumeSpecName "kube-api-access-4xkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:28.096737 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.096719    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca" pod "0b9dc441-8bc0-421b-866f-f509478a57dd" (UID: "0b9dc441-8bc0-421b-866f-f509478a57dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:28.195937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.195868    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9dc441-8bc0-421b-866f-f509478a57dd-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:28.195937 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.195898    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xkm8\" (UniqueName: \"kubernetes.io/projected/0b9dc441-8bc0-421b-866f-f509478a57dd-kube-api-access-4xkm8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:28.793633 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.793594    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn" event={"ID":"83c307a8-ac72-41f6-8cd4-234f1368933b","Type":"ContainerStarted","Data":"52bdd2b9d447a4871af47d6aca8ed7d3a2397a713655b50802ba00e2405b88e0"}
Oct 30 02:57:28.793829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.793691    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn" event={"ID":"83c307a8-ac72-41f6-8cd4-234f1368933b","Type":"ContainerStarted","Data":"155b03b93c44d0fc24bb61e28a96d36e3df6751d0174f175d7474be02e1bf044"}
Oct 30 02:57:28.795000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.794970    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"
Oct 30 02:57:28.795124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.795010    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4" event={"ID":"0b9dc441-8bc0-421b-866f-f509478a57dd","Type":"ContainerDied","Data":"733c3d22ccda8f4975121406e49faf6cd272393ce2b6e4e2011949cbe24fbd7d"}
Oct 30 02:57:28.795124 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.795051    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733c3d22ccda8f4975121406e49faf6cd272393ce2b6e4e2011949cbe24fbd7d"
Oct 30 02:57:28.846653 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:28.846602    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn" podStartSLOduration=1.846579162 podStartE2EDuration="1.846579162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:28.842814961 +0000 UTC m=+987.916985755" watchObservedRunningTime="2026-10-30 02:57:28.846579162 +0000 UTC m=+987.920749958"
Oct 30 02:57:29.034921 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.034886    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 02:57:29.035868 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.035850    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0b9dc441-8bc0-421b-866f-f509478a57dd" containerName="revoke-keys"
Oct 30 02:57:29.035922 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.035871    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9dc441-8bc0-421b-866f-f509478a57dd" containerName="revoke-keys"
Oct 30 02:57:29.036029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.036018    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9dc441-8bc0-421b-866f-f509478a57dd" containerName="revoke-keys"
Oct 30 02:57:29.044373 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.044304    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.071090 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.071063    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 02:57:29.107638 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.107606    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.107788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.107676    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.107788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.107759    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.107899 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.107801    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.107899 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.107833    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.108035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.108017    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.108100 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.108049    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm575\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.108159 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.108133    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.108266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.108220    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.181531 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.181496    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 02:57:29.186212 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.186190    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.209669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.209638    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.209796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.209689    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.209863 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.209835    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.209979 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.209960    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.209994    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210023    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210078    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210263    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210273    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210279    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210334    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210412    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210467    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210520    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210548    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210581    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210613    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxmv\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210711    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210745    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm575\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210800    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210828    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210860    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.210965    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.213251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.211085    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.214396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.214213    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.214689 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.214654    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.219996 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.219968    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 02:57:29.226905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.226389    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.249620 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.249593    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 02:57:29.252270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.251708    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.252270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.251753    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm575\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.312164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312099    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312164 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312146    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312197    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312221    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312386 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312328    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxmv\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312416    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312447    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312476    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312534    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312541    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312572    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312579    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312679    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bm8v\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312713    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.312751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312751    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.313014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312805    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.313014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312804    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.313014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312861    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.313014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312942    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.313014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.312973    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.313243 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.313092    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.313243 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.313146    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.313422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.313403    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.315126 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.315098    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.315431 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.315412    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.337082 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.337059    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.337526 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.337506    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxmv\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.369079 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.369053    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:29.414910 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.414875    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.414946    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415075 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415020    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415079    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415132    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bm8v\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415159    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415372 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415290    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415372 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415317    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415476 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415391    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415753 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415702    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415760    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415831    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415842    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.415973 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.415875    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.417637 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.417614    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.418053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.418033    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.440379 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.440353    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.443432 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.441301    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bm8v\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.506014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.505517    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:29.585450 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.585415    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:29.692758 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:29.692733    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49c6ffe_a90e_4cfb_9322_22abdaedea6c.slice/crio-86168fb4f5e8f84193d1fcc8fb9503313a0f67918989a2bf4c446b58f0ef76c0 WatchSource:0}: Error finding container 86168fb4f5e8f84193d1fcc8fb9503313a0f67918989a2bf4c446b58f0ef76c0: Status 404 returned error can't find the container with id 86168fb4f5e8f84193d1fcc8fb9503313a0f67918989a2bf4c446b58f0ef76c0
Oct 30 02:57:29.695504 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.695468    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.695599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.695535    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.695599 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.695583    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.701054 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.701029    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 02:57:29.806720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.806683    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" event={"ID":"e49c6ffe-a90e-4cfb-9322-22abdaedea6c","Type":"ContainerStarted","Data":"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"}
Oct 30 02:57:29.807081 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.806755    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" event={"ID":"e49c6ffe-a90e-4cfb-9322-22abdaedea6c","Type":"ContainerStarted","Data":"86168fb4f5e8f84193d1fcc8fb9503313a0f67918989a2bf4c446b58f0ef76c0"}
Oct 30 02:57:29.808847 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.808819    4214 generic.go:356] "Generic (PLEG): container finished" podID="83c307a8-ac72-41f6-8cd4-234f1368933b" containerID="52bdd2b9d447a4871af47d6aca8ed7d3a2397a713655b50802ba00e2405b88e0" exitCode=0
Oct 30 02:57:29.808959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.808948    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn" event={"ID":"83c307a8-ac72-41f6-8cd4-234f1368933b","Type":"ContainerDied","Data":"52bdd2b9d447a4871af47d6aca8ed7d3a2397a713655b50802ba00e2405b88e0"}
Oct 30 02:57:29.893984 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.893047    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podStartSLOduration=0.893030334 podStartE2EDuration="893.030334ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:29.863709342 +0000 UTC m=+988.937880136" watchObservedRunningTime="2026-10-30 02:57:29.893030334 +0000 UTC m=+988.967201129"
Oct 30 02:57:29.895615 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.895589    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 02:57:29.896279 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:29.896212    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733d454d_eeae_4735_9665_6d42e983e61f.slice/crio-102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5 WatchSource:0}: Error finding container 102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5: Status 404 returned error can't find the container with id 102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5
Oct 30 02:57:29.899251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.899210    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.899335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.899291    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.899335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.899322    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.927352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.927328    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 02:57:29.952169 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.952132    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.952532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.952508    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.952765 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.952740    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:29.971813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:29.971785    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 02:57:30.002341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.002320    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.006246 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.006040    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029369    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029449    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029470    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029487    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029510    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029545    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029613    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029634    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.030661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.029660    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc978\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.134730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.134668    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.143384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.143358    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.144196 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.144045    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.144517 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.144499    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc978\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.144780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.144667    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.144922 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.135718    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.145136 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.145117    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.145608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.145589    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.145939 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.145915    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.146043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.145958    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.146043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.145986    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.146425 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.146403    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.147628 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.147605    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.148486 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.148025    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.161028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.160993    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.162025 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.161548    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.169550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.169531    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc978\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.170141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.170094    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.345254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.340508    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:30.370468 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.370150    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:30.372547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.372056    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:30.372547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.372108    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:30.826399 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.825275    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 02:57:30.830420 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.830383    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:30.830523 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.830451    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:30.830523 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.830481    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:30.867251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.866647    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" event={"ID":"3ea50e0f-8890-4576-b68b-27a43b7073b9","Type":"ContainerStarted","Data":"69df8969e17124c9d68514d14b7ae2b66dd4dd1e0539bc678b08d587404a13c7"}
Oct 30 02:57:30.871251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.869096    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" event={"ID":"c2ce4243-031c-4132-a5e8-946854676d44","Type":"ContainerStarted","Data":"6bf4c15f985337f54589d9307d3569535151f56096c9406592c7441c5373c855"}
Oct 30 02:57:30.871251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.869152    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" event={"ID":"c2ce4243-031c-4132-a5e8-946854676d44","Type":"ContainerStarted","Data":"ed8881dbeb43f44dc4d1161410e901ed8069a427a438de26f808d2bf9d5b9461"}
Oct 30 02:57:30.879216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.878403    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" event={"ID":"733d454d-eeae-4735-9665-6d42e983e61f","Type":"ContainerStarted","Data":"7b3906d400e780300fc60047165a8ff20e915dc32d78736803d0027402695836"}
Oct 30 02:57:30.879216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:30.878454    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" event={"ID":"733d454d-eeae-4735-9665-6d42e983e61f","Type":"ContainerStarted","Data":"102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5"}
Oct 30 02:57:31.093308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.092674    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podStartSLOduration=2.092651671 podStartE2EDuration="2.092651671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:30.919869377 +0000 UTC m=+989.994040172" watchObservedRunningTime="2026-10-30 02:57:31.092651671 +0000 UTC m=+990.166822464"
Oct 30 02:57:31.160887 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.160831    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podStartSLOduration=2.160812394 podStartE2EDuration="2.160812394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:31.014554652 +0000 UTC m=+990.088725445" watchObservedRunningTime="2026-10-30 02:57:31.160812394 +0000 UTC m=+990.234983190"
Oct 30 02:57:31.408883 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.408795    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.409143 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.409111    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:31.538652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.531294    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:31.539754 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.539724    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.550373 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.550333    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:31.654691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.654595    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:31.860672 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.860639    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.879863 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:31.879813    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:32.328247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.309972    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" event={"ID":"3ea50e0f-8890-4576-b68b-27a43b7073b9","Type":"ContainerStarted","Data":"83e8fbbdfc96cb8499bc3f44b3b16875519ed1dbb5b873da931bee68e6d41823"}
Oct 30 02:57:32.404266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.395291    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:32.404266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.398413    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.404266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.398466    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:32.404266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.398890    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.404266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.398928    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:32.426097 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.426018    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podStartSLOduration=3.425994807 podStartE2EDuration="3.425994807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:32.402446695 +0000 UTC m=+991.476617526" watchObservedRunningTime="2026-10-30 02:57:32.425994807 +0000 UTC m=+991.500165601"
Oct 30 02:57:32.455058 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.455028    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:32.558732 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.558695    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.558978 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.558951    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:32.589282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.586591    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.589282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.586655    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:32.589282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.587822    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\" (UniqueName: \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\") pod \"83c307a8-ac72-41f6-8cd4-234f1368933b\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") "
Oct 30 02:57:32.589282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.587938    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\") pod \"83c307a8-ac72-41f6-8cd4-234f1368933b\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") "
Oct 30 02:57:32.589282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.588091    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\") pod \"83c307a8-ac72-41f6-8cd4-234f1368933b\" (UID: \"83c307a8-ac72-41f6-8cd4-234f1368933b\") "
Oct 30 02:57:32.602275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.599415    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca" pod "83c307a8-ac72-41f6-8cd4-234f1368933b" (UID: "83c307a8-ac72-41f6-8cd4-234f1368933b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:32.737253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.694633    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:32.763339 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.763257    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg" pod "83c307a8-ac72-41f6-8cd4-234f1368933b" (UID: "83c307a8-ac72-41f6-8cd4-234f1368933b"). InnerVolumeSpecName "kube-api-access-7z7rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:32.787479 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.787402    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca" pod "83c307a8-ac72-41f6-8cd4-234f1368933b" (UID: "83c307a8-ac72-41f6-8cd4-234f1368933b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:32.876353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.855909    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7rg\" (UniqueName: \"kubernetes.io/projected/83c307a8-ac72-41f6-8cd4-234f1368933b-kube-api-access-7z7rg\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:32.876353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:32.855945    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c307a8-ac72-41f6-8cd4-234f1368933b-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:33.364251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.360497    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.364251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.360554    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:33.376254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.369569    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.376254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.369627    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:33.446050 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.446013    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn" event={"ID":"83c307a8-ac72-41f6-8cd4-234f1368933b","Type":"ContainerDied","Data":"155b03b93c44d0fc24bb61e28a96d36e3df6751d0174f175d7474be02e1bf044"}
Oct 30 02:57:33.446338 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.446318    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="155b03b93c44d0fc24bb61e28a96d36e3df6751d0174f175d7474be02e1bf044"
Oct 30 02:57:33.446518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.446505    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"
Oct 30 02:57:33.514732 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.514691    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:33.524752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.524717    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.525964 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.525930    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:33.589320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.587183    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.589320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.587262    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:33.939918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:33.939794    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:34.107439 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.107393    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:34.269549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.269454    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"]
Oct 30 02:57:34.333610 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.333580    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83c307a8-ac72-41f6-8cd4-234f1368933b" containerName="revoke-keys"
Oct 30 02:57:34.333990 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.333974    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c307a8-ac72-41f6-8cd4-234f1368933b" containerName="revoke-keys"
Oct 30 02:57:34.335085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.334385    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c307a8-ac72-41f6-8cd4-234f1368933b" containerName="revoke-keys"
Oct 30 02:57:34.346341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.346032    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.346341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.346087    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:34.356727 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.356696    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"]
Oct 30 02:57:34.357127 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.357022    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.360563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.359760    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:34.377984 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.377955    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:34.384717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.384602    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:34.385280 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.385216    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.400134 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.385437    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:34.521542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.521436    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:34.525283 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.522827    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:34.535263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.531331    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.535263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.531377    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.535263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.531405    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfrz\" (UniqueName: \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.535263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.531720    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.535263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.531764    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:34.587068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.587013    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.587378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.587333    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:34.636305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.632933    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.656377 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.656331    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.656625 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.656605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfrz\" (UniqueName: \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:34.827312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:34.658106    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:35.182322 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.116552    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfrz\" (UniqueName: \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:35.307263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.297510    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:35.307263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.306025    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde7b2fd-4bxkk\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:35.372785 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.372751    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:35.390961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.390925    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.391992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.391941    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:35.392274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.391324    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.392427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.392407    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:35.525539 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.524573    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:35.632342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.618402    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.632342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.618473    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:35.661283 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.641441    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.661283 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:35.641505    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:36.063357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.057048    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:36.357704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.357667    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.357958 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.357930    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:36.376410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.376364    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:36.377300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.377274    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.402376 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.402329    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:36.496302 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.494926    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:36.636831 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.636739    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.637097 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.637064    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:36.675252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.660850    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.675252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.660903    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:36.940985 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:36.935820    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:37.162256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.112799    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:57:37.529812 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.529712    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:37.798076 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.798039    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:37.959293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.799512    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:57:37.960817 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.957828    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:57:37.960981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.883465    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:37.961095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.956545    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:37.961205 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.961182    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:38.035695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:37.959254    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:38.080448 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.079084    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:38.162262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.160048    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:38.162262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.139613    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:38.202309 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:57:38.181393    4214 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.075s"
Oct 30 02:57:38.202629 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.202603    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:38.202764 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.202750    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"]
Oct 30 02:57:38.202867 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.202830    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:38.215541 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.215509    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"]
Oct 30 02:57:38.221299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.221262    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:38.239459 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.239427    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"]
Oct 30 02:57:38.240055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.240032    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk" event={"ID":"60948ee3-4f25-4fc4-a82f-60353bba50ab","Type":"ContainerStarted","Data":"76bc31d7c3d70bf80274de079b2882ad65d6472060ebd977c4dbf9a9c2085e8e"}
Oct 30 02:57:38.240207 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.239977    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.301795 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.301765    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:38.318247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.318209    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:38.432314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.423361    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:38.432314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.423416    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:38.433470 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.433437    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 02:57:38.433686 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.433655    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 02:57:38.439259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.434645    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75x2\" (UniqueName: \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.439259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.434914    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.439259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.434999    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.514088 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.514054    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:38.515359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.515330    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:38.597662 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.596994    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.598312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.598281    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.598491 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.598472    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75x2\" (UniqueName: \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.599961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.599937    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.602521 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.601280    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:38.602674 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.602650    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:38.615788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.615758    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.624869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.624836    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75x2\" (UniqueName: \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\") pod \"maas-api-revoke-sub-ed63bde4-bcnz7\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:38.625299 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:38.625279    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:39.201646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.201594    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"]
Oct 30 02:57:39.301333 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.301293    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7" event={"ID":"ae80dad1-fc2d-4271-b99a-babf572f1bb9","Type":"ContainerStarted","Data":"5f0db213e6a6c1f27553ec8153c035efd0fd95a7f5ae0db10bc28d28ecef9e4f"}
Oct 30 02:57:39.302855 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:57:39.302824    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:57:39.304538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.304507    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk" event={"ID":"60948ee3-4f25-4fc4-a82f-60353bba50ab","Type":"ContainerStarted","Data":"133aa75f69fc44a970a0a59d16888f8d29bb6a695b61150ef2821d319a5fb53d"}
Oct 30 02:57:39.352807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.345553    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:39.352807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.345616    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:39.352807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.347334    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk" podStartSLOduration=5.347316414 podStartE2EDuration="5.347316414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:39.346297715 +0000 UTC m=+998.420468510" watchObservedRunningTime="2026-10-30 02:57:39.347316414 +0000 UTC m=+998.421487208"
Oct 30 02:57:39.510405 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.510027    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 02:57:39.510405 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.510090    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 02:57:39.606049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.605624    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 02:57:39.606049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:39.605680    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 02:57:40.350264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.345612    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:40.350264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.345672    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:40.390265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.386482    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:40.390265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.386566    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:40.397617 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.397578    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7" event={"ID":"ae80dad1-fc2d-4271-b99a-babf572f1bb9","Type":"ContainerStarted","Data":"65b94f994a485670255a7694cdd91a62d254b4a339223ca962b74e0594ef22e5"}
Oct 30 02:57:40.426251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.419749    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk" event={"ID":"60948ee3-4f25-4fc4-a82f-60353bba50ab","Type":"ContainerDied","Data":"133aa75f69fc44a970a0a59d16888f8d29bb6a695b61150ef2821d319a5fb53d"}
Oct 30 02:57:40.426251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.413997    4214 generic.go:356] "Generic (PLEG): container finished" podID="60948ee3-4f25-4fc4-a82f-60353bba50ab" containerID="133aa75f69fc44a970a0a59d16888f8d29bb6a695b61150ef2821d319a5fb53d" exitCode=0
Oct 30 02:57:40.550314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.549793    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7" podStartSLOduration=3.549770886 podStartE2EDuration="3.549770886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:40.51208475 +0000 UTC m=+999.586255544" watchObservedRunningTime="2026-10-30 02:57:40.549770886 +0000 UTC m=+999.623941696"
Oct 30 02:57:40.558957 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.558924    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:40.560343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.560318    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:40.582258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:40.580775    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 02:57:41.349253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.346655    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:41.349253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.346720    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:41.520277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.506524    4214 generic.go:356] "Generic (PLEG): container finished" podID="ae80dad1-fc2d-4271-b99a-babf572f1bb9" containerID="65b94f994a485670255a7694cdd91a62d254b4a339223ca962b74e0594ef22e5" exitCode=0
Oct 30 02:57:41.520277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.508134    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7" event={"ID":"ae80dad1-fc2d-4271-b99a-babf572f1bb9","Type":"ContainerDied","Data":"65b94f994a485670255a7694cdd91a62d254b4a339223ca962b74e0594ef22e5"}
Oct 30 02:57:41.540253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.535316    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:41.540253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.535379    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:41.593363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.589157    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:41.593363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.589218    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:41.951259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:41.949357    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.317578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.315756    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.537875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.537346    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.552276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.539298    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 02:57:42.552276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.539346    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 02:57:42.553034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.552595    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:42.553034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.552648    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.600427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.600328    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:42.600668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.600637    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.979251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.978747    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.987251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:42.980150    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:57:43.007247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.001543    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:43.236957 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.236875    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.307780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.295803    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\" (UniqueName: \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\") pod \"60948ee3-4f25-4fc4-a82f-60353bba50ab\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") "
Oct 30 02:57:43.307780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.295944    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\") pod \"60948ee3-4f25-4fc4-a82f-60353bba50ab\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") "
Oct 30 02:57:43.307780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.296099    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\") pod \"60948ee3-4f25-4fc4-a82f-60353bba50ab\" (UID: \"60948ee3-4f25-4fc4-a82f-60353bba50ab\") "
Oct 30 02:57:43.357423 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.328589    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca" pod "60948ee3-4f25-4fc4-a82f-60353bba50ab" (UID: "60948ee3-4f25-4fc4-a82f-60353bba50ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:43.357423 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.328874    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.357423 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.330207    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:43.439253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.430034    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:43.721263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.652595    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz" pod "60948ee3-4f25-4fc4-a82f-60353bba50ab" (UID: "60948ee3-4f25-4fc4-a82f-60353bba50ab"). InnerVolumeSpecName "kube-api-access-8hfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:43.721263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.652862    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.721263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.654391    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca" pod "60948ee3-4f25-4fc4-a82f-60353bba50ab" (UID: "60948ee3-4f25-4fc4-a82f-60353bba50ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:43.721263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.656194    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfrz\" (UniqueName: \"kubernetes.io/projected/60948ee3-4f25-4fc4-a82f-60353bba50ab-kube-api-access-8hfrz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:43.817875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.734524    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:43.870218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.869519    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:43.870218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.869585    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.944256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.656219    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60948ee3-4f25-4fc4-a82f-60353bba50ab-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.000463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:43.950584    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:44.014364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.014314    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.021766 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.021724    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.023944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.023910    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fmtkt" podUID="04056afb-ee76-4d83-a900-95fc26f11e7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.259308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.237740    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.259308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.238637    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:44.357260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.353622    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.443259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.442729    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:44.443259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.442804    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.486261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.456884    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk" event={"ID":"60948ee3-4f25-4fc4-a82f-60353bba50ab","Type":"ContainerDied","Data":"76bc31d7c3d70bf80274de079b2882ad65d6472060ebd977c4dbf9a9c2085e8e"}
Oct 30 02:57:44.486261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.456954    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76bc31d7c3d70bf80274de079b2882ad65d6472060ebd977c4dbf9a9c2085e8e"
Oct 30 02:57:44.486261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.457029    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"
Oct 30 02:57:44.534997 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.526965    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:44.590402 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.590370    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:44.728254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.703541    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.993580 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.993547    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:44.993771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:44.993743    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.023528 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.023487    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.253314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.252666    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:45.253314 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.252745    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.331298 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.315059    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 02:57:45.417576 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.417522    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.540436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.540335    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:57:45.621041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.620989    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:45.622342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.622305    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.694291 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.680910    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:45.786194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.786153    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:45.838772 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.838735    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:45.935277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.932922    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"
Oct 30 02:57:45.935808 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.935678    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7" event={"ID":"ae80dad1-fc2d-4271-b99a-babf572f1bb9","Type":"ContainerDied","Data":"5f0db213e6a6c1f27553ec8153c035efd0fd95a7f5ae0db10bc28d28ecef9e4f"}
Oct 30 02:57:45.936108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:45.936080    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0db213e6a6c1f27553ec8153c035efd0fd95a7f5ae0db10bc28d28ecef9e4f"
Oct 30 02:57:46.031369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.031316    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:46.054871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.054419    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:46.054871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.054493    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:46.055945 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.055914    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\") pod \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") "
Oct 30 02:57:46.056082 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.056044    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\") pod \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") "
Oct 30 02:57:46.056306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.056284    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\" (UniqueName: \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\") pod \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\" (UID: \"ae80dad1-fc2d-4271-b99a-babf572f1bb9\") "
Oct 30 02:57:46.092909 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.092808    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca" pod "ae80dad1-fc2d-4271-b99a-babf572f1bb9" (UID: "ae80dad1-fc2d-4271-b99a-babf572f1bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:46.132292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.130421    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca" pod "ae80dad1-fc2d-4271-b99a-babf572f1bb9" (UID: "ae80dad1-fc2d-4271-b99a-babf572f1bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:46.170740 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.170702    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:46.171005 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.170974    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae80dad1-fc2d-4271-b99a-babf572f1bb9-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:46.245255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.222091    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:57:46.245255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.223779    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2" pod "ae80dad1-fc2d-4271-b99a-babf572f1bb9" (UID: "ae80dad1-fc2d-4271-b99a-babf572f1bb9"). InnerVolumeSpecName "kube-api-access-j75x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:46.276086 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.275385    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75x2\" (UniqueName: \"kubernetes.io/projected/ae80dad1-fc2d-4271-b99a-babf572f1bb9-kube-api-access-j75x2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:46.350529 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.350435    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:46.350788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.350756    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:46.503024 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.495580    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:57:46.521754 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:57:46.521665    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60948ee3_4f25_4fc4_a82f_60353bba50ab.slice/crio-76bc31d7c3d70bf80274de079b2882ad65d6472060ebd977c4dbf9a9c2085e8e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60948ee3_4f25_4fc4_a82f_60353bba50ab.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 02:57:46.553770 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.544741    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:46.725410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.725319    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:46.733263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.725629    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:46.757597 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.757568    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:46.794122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:46.794093    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:57:47.117398 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.117363    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:47.117650 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.117621    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:47.365255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.337975    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"]
Oct 30 02:57:47.384306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.366478    4214 cpu_manager.go:414] "containerMap: removing container" podUID="60948ee3-4f25-4fc4-a82f-60353bba50ab" containerName="revoke-keys"
Oct 30 02:57:47.384511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.384485    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60948ee3-4f25-4fc4-a82f-60353bba50ab" containerName="revoke-keys"
Oct 30 02:57:47.429956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.429927    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ae80dad1-fc2d-4271-b99a-babf572f1bb9" containerName="revoke-keys"
Oct 30 02:57:47.430166 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.430151    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae80dad1-fc2d-4271-b99a-babf572f1bb9" containerName="revoke-keys"
Oct 30 02:57:47.430515 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.430499    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae80dad1-fc2d-4271-b99a-babf572f1bb9" containerName="revoke-keys"
Oct 30 02:57:47.430609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.430600    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="60948ee3-4f25-4fc4-a82f-60353bba50ab" containerName="revoke-keys"
Oct 30 02:57:47.448264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.419680    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:47.448264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.434320    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:47.484681 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.419748    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:47.535690 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.535650    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"]
Oct 30 02:57:47.535992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.535973    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 02:57:47.536273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.536255    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.551255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.549811    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:47.564620 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.554334    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:57:47.564620 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.554403    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:57:47.583804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.577979    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.583804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.578047    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.583804 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.578174    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpk9\" (UniqueName: \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.733646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.733574    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:57:47.736997 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.736965    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.737924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.737902    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.738195 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.738178    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpk9\" (UniqueName: \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.739059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.737834    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.787793 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.787761    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:47.961046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.958888    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpk9\" (UniqueName: \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:47.961046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:47.959212    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 02:57:48.056203 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:48.039116    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-963ff370-m5t54\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:48.096127 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:48.096098    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:48.127059 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:48.126989    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 02:57:48.334485 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:48.334405    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:48.646334 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:48.646291    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb542574_187d_48ec_94a3_2241ea14b3b8.slice/crio-1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4 WatchSource:0}: Error finding container 1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4: Status 404 returned error can't find the container with id 1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4
Oct 30 02:57:48.647397 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:48.647372    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"]
Oct 30 02:57:49.343887 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:49.343827    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54" event={"ID":"cb542574-187d-48ec-94a3-2241ea14b3b8","Type":"ContainerStarted","Data":"e1f7152e1a8149794afd73ae5f479ec0e47db68de2d8ebca855cd53a9567e0ba"}
Oct 30 02:57:49.344390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:49.343901    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54" event={"ID":"cb542574-187d-48ec-94a3-2241ea14b3b8","Type":"ContainerStarted","Data":"1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4"}
Oct 30 02:57:50.356012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:50.355976    4214 generic.go:356] "Generic (PLEG): container finished" podID="cb542574-187d-48ec-94a3-2241ea14b3b8" containerID="e1f7152e1a8149794afd73ae5f479ec0e47db68de2d8ebca855cd53a9567e0ba" exitCode=0
Oct 30 02:57:50.356548 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:50.356034    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54" event={"ID":"cb542574-187d-48ec-94a3-2241ea14b3b8","Type":"ContainerDied","Data":"e1f7152e1a8149794afd73ae5f479ec0e47db68de2d8ebca855cd53a9567e0ba"}
Oct 30 02:57:51.554391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.554360    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:51.643477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.643382    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\") pod \"cb542574-187d-48ec-94a3-2241ea14b3b8\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") "
Oct 30 02:57:51.643639 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.643611    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\" (UniqueName: \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\") pod \"cb542574-187d-48ec-94a3-2241ea14b3b8\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") "
Oct 30 02:57:51.643713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.643667    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\") pod \"cb542574-187d-48ec-94a3-2241ea14b3b8\" (UID: \"cb542574-187d-48ec-94a3-2241ea14b3b8\") "
Oct 30 02:57:51.643849 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.643823    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca" pod "cb542574-187d-48ec-94a3-2241ea14b3b8" (UID: "cb542574-187d-48ec-94a3-2241ea14b3b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:51.644211 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.644181    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:51.646380 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.646353    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9" pod "cb542574-187d-48ec-94a3-2241ea14b3b8" (UID: "cb542574-187d-48ec-94a3-2241ea14b3b8"). InnerVolumeSpecName "kube-api-access-nvpk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:51.646475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.646443    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca" pod "cb542574-187d-48ec-94a3-2241ea14b3b8" (UID: "cb542574-187d-48ec-94a3-2241ea14b3b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:51.746000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.745956    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpk9\" (UniqueName: \"kubernetes.io/projected/cb542574-187d-48ec-94a3-2241ea14b3b8-kube-api-access-nvpk9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:51.746000 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:51.745997    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb542574-187d-48ec-94a3-2241ea14b3b8-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:52.090076 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.090040    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"]
Oct 30 02:57:52.090816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.090794    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cb542574-187d-48ec-94a3-2241ea14b3b8" containerName="revoke-keys"
Oct 30 02:57:52.090816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.090815    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb542574-187d-48ec-94a3-2241ea14b3b8" containerName="revoke-keys"
Oct 30 02:57:52.090944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.090921    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb542574-187d-48ec-94a3-2241ea14b3b8" containerName="revoke-keys"
Oct 30 02:57:52.094786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.094768    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.116374 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.116341    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"]
Oct 30 02:57:52.254646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.254575    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.254646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.254638    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.255251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.254913    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v2p\" (UniqueName: \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.356041 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.355951    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v2p\" (UniqueName: \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.356191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.356105    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.356191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.356141    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.357048 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.357019    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.358799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.358773    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.377819 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.377793    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v2p\" (UniqueName: \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\") pod \"maas-api-revoke-sub-527922ba-s6b66\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.379160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.379137    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 02:57:52.379160 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.379135    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54" event={"ID":"cb542574-187d-48ec-94a3-2241ea14b3b8","Type":"ContainerDied","Data":"1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4"}
Oct 30 02:57:52.379406 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.379189    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad3c28eadefe5847d4fa541c8819943b539384be0b2b1003418189d598e77e4"
Oct 30 02:57:52.408927 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.408893    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:52.743069 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:57:52.743032    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca90fa5_4f50_4699_bb74_064514822211.slice/crio-4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5 WatchSource:0}: Error finding container 4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5: Status 404 returned error can't find the container with id 4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5
Oct 30 02:57:52.745750 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:52.745722    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"]
Oct 30 02:57:53.389846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:53.389811    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66" event={"ID":"5ca90fa5-4f50-4699-bb74-064514822211","Type":"ContainerStarted","Data":"3f92d3f4786c20d121ad79d1711d5a13f045d93358d201f192e748d2c345a712"}
Oct 30 02:57:53.389989 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:53.389866    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66" event={"ID":"5ca90fa5-4f50-4699-bb74-064514822211","Type":"ContainerStarted","Data":"4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5"}
Oct 30 02:57:53.431783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:53.431689    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66" podStartSLOduration=1.431674753 podStartE2EDuration="1.431674753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:53.428802651 +0000 UTC m=+1012.502973455" watchObservedRunningTime="2026-10-30 02:57:53.431674753 +0000 UTC m=+1012.505845545"
Oct 30 02:57:54.402292 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:54.402210    4214 generic.go:356] "Generic (PLEG): container finished" podID="5ca90fa5-4f50-4699-bb74-064514822211" containerID="3f92d3f4786c20d121ad79d1711d5a13f045d93358d201f192e748d2c345a712" exitCode=0
Oct 30 02:57:54.403010 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:54.402959    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66" event={"ID":"5ca90fa5-4f50-4699-bb74-064514822211","Type":"ContainerDied","Data":"3f92d3f4786c20d121ad79d1711d5a13f045d93358d201f192e748d2c345a712"}
Oct 30 02:57:55.834550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:55.834402    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:55.960250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:55.938636    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\") pod \"5ca90fa5-4f50-4699-bb74-064514822211\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") "
Oct 30 02:57:55.960250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:55.959422    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\" (UniqueName: \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\") pod \"5ca90fa5-4f50-4699-bb74-064514822211\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") "
Oct 30 02:57:55.960250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:55.959502    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\") pod \"5ca90fa5-4f50-4699-bb74-064514822211\" (UID: \"5ca90fa5-4f50-4699-bb74-064514822211\") "
Oct 30 02:57:55.995771 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:55.960594    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca" pod "5ca90fa5-4f50-4699-bb74-064514822211" (UID: "5ca90fa5-4f50-4699-bb74-064514822211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:56.003333 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.003297    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca" pod "5ca90fa5-4f50-4699-bb74-064514822211" (UID: "5ca90fa5-4f50-4699-bb74-064514822211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:56.045307 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.038844    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p" pod "5ca90fa5-4f50-4699-bb74-064514822211" (UID: "5ca90fa5-4f50-4699-bb74-064514822211"). InnerVolumeSpecName "kube-api-access-j2v2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:56.075264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.068176    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v2p\" (UniqueName: \"kubernetes.io/projected/5ca90fa5-4f50-4699-bb74-064514822211-kube-api-access-j2v2p\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.075873 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.075847    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.076035 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.076021    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca90fa5-4f50-4699-bb74-064514822211-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:57:56.459282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.459244    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66" event={"ID":"5ca90fa5-4f50-4699-bb74-064514822211","Type":"ContainerDied","Data":"4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5"}
Oct 30 02:57:56.459509 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.459496    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ccf1647a557d594ab330d545efbd9bc96939f7fe404e34c0f6935f6f1921cf5"
Oct 30 02:57:56.459646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:56.459636    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"
Oct 30 02:57:57.295084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:57.295034    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:59.234255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.233317    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:59.571045 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.571001    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 02:57:59.580324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.572511    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5ca90fa5-4f50-4699-bb74-064514822211" containerName="revoke-keys"
Oct 30 02:57:59.580583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.580554    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca90fa5-4f50-4699-bb74-064514822211" containerName="revoke-keys"
Oct 30 02:57:59.606279 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.580963    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca90fa5-4f50-4699-bb74-064514822211" containerName="revoke-keys"
Oct 30 02:57:59.618261 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.612482    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.666417 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.666367    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:59.678344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.678308    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688327    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688372    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688470    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688684    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688726    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688766    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688807    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688871    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.689269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.688898    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfl4\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.801781 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.801742    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.801988 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.801960    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802024    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802074    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802145    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802178    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfl4\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802332 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802221    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802332 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802290    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.802430 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802379    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.804249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.802894    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.804249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.803213    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.804249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.804129    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.804758 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.804718    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.804846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.804781    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.829860 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.827996    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfl4\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.830664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.830475    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.832416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.832391    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:57:59.833594 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:57:59.833566    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:00.036603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.036559    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:00.370437 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.370409    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 02:58:00.373903 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:00.373866    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1d97f4_1c98_4fce_951b_ac3de38bf081.slice/crio-c31f0e5afe51671a58c804c1fc62abf6742c8a04cc021a2d033a1f319b99b1ad WatchSource:0}: Error finding container c31f0e5afe51671a58c804c1fc62abf6742c8a04cc021a2d033a1f319b99b1ad: Status 404 returned error can't find the container with id c31f0e5afe51671a58c804c1fc62abf6742c8a04cc021a2d033a1f319b99b1ad
Oct 30 02:58:00.377070 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.377033    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:00.377163 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.377119    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:00.377208 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.377183    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:58:00.734880 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.734775    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" event={"ID":"cb1d97f4-1c98-4fce-951b-ac3de38bf081","Type":"ContainerStarted","Data":"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"}
Oct 30 02:58:00.735050 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.734881    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" event={"ID":"cb1d97f4-1c98-4fce-951b-ac3de38bf081","Type":"ContainerStarted","Data":"c31f0e5afe51671a58c804c1fc62abf6742c8a04cc021a2d033a1f319b99b1ad"}
Oct 30 02:58:00.795141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:00.795081    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podStartSLOduration=1.7950656 podStartE2EDuration="1.7950656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:00.79019167 +0000 UTC m=+1019.864362458" watchObservedRunningTime="2026-10-30 02:58:00.7950656 +0000 UTC m=+1019.869236392"
Oct 30 02:58:01.037069 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:01.036982    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:01.038516 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:01.038493    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:01.038637 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:01.038541    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:02.041040 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.041004    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:02.041684 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.041637    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:02.202664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.202628    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"]
Oct 30 02:58:02.208521 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.208500    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.225817 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.225638    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"]
Oct 30 02:58:02.232363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.232319    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:02.232479 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.232389    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:02.296357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.296321    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4zl\" (UniqueName: \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.296609 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.296587    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.296701 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.296630    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.399832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.399785    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.400009 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.399862    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.400095 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.400074    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4zl\" (UniqueName: \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.405149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.405115    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.426839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.425781    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.499901 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.499854    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4zl\" (UniqueName: \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\") pod \"maas-api-revoke-sub-d0b3c61a-hzqhs\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:02.523185 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:02.523156    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:03.042547 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:03.042440    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:03.043207 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:03.043175    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:03.385357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:03.385322    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"]
Oct 30 02:58:03.884368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:03.884211    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs" event={"ID":"5b3018a4-a1be-4bc3-a528-c0e27a2329b2","Type":"ContainerStarted","Data":"6cc91cbf5e8084db81a504c2226a593dac9759a11a3b2d61c32b5700e129c56c"}
Oct 30 02:58:03.932278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:03.932179    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs" podStartSLOduration=1.932165501 podStartE2EDuration="1.932165501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:03.927080748 +0000 UTC m=+1023.001251543" watchObservedRunningTime="2026-10-30 02:58:03.932165501 +0000 UTC m=+1023.006336320"
Oct 30 02:58:04.038828 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:04.038794    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:04.039085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:04.039059    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:04.893796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:04.893760    4214 generic.go:356] "Generic (PLEG): container finished" podID="5b3018a4-a1be-4bc3-a528-c0e27a2329b2" containerID="8f2883131b0fb34c140e1e6d93711fe8e4de2ed86a629d8b140f8579506b5d84" exitCode=0
Oct 30 02:58:04.894194 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:04.893842    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs" event={"ID":"5b3018a4-a1be-4bc3-a528-c0e27a2329b2","Type":"ContainerDied","Data":"8f2883131b0fb34c140e1e6d93711fe8e4de2ed86a629d8b140f8579506b5d84"}
Oct 30 02:58:05.063253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:05.058440    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:05.063253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:05.058504    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:06.037578 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.037542    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:06.038129 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.038094    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:06.119007 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.118985    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:06.294755 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.294723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\") pod \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") "
Oct 30 02:58:06.294939 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.294877    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\" (UniqueName: \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\") pod \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") "
Oct 30 02:58:06.294939 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.294915    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\") pod \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\" (UID: \"5b3018a4-a1be-4bc3-a528-c0e27a2329b2\") "
Oct 30 02:58:06.295336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.295306    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca" pod "5b3018a4-a1be-4bc3-a528-c0e27a2329b2" (UID: "5b3018a4-a1be-4bc3-a528-c0e27a2329b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:06.295727 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.295706    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:06.296842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.296817    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca" pod "5b3018a4-a1be-4bc3-a528-c0e27a2329b2" (UID: "5b3018a4-a1be-4bc3-a528-c0e27a2329b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:06.297039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.297016    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl" pod "5b3018a4-a1be-4bc3-a528-c0e27a2329b2" (UID: "5b3018a4-a1be-4bc3-a528-c0e27a2329b2"). InnerVolumeSpecName "kube-api-access-6n4zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:06.396610 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.396570    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:06.396610 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.396608    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4zl\" (UniqueName: \"kubernetes.io/projected/5b3018a4-a1be-4bc3-a528-c0e27a2329b2-kube-api-access-6n4zl\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:06.911767 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.911735    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"
Oct 30 02:58:06.911975 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.911765    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs" event={"ID":"5b3018a4-a1be-4bc3-a528-c0e27a2329b2","Type":"ContainerDied","Data":"6cc91cbf5e8084db81a504c2226a593dac9759a11a3b2d61c32b5700e129c56c"}
Oct 30 02:58:06.911975 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:06.911849    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc91cbf5e8084db81a504c2226a593dac9759a11a3b2d61c32b5700e129c56c"
Oct 30 02:58:07.037222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:07.037187    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 02:58:07.037383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:07.037265    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 02:58:08.445561 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:08.445527    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:08.447141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:08.446379    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:08.448052 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:08.448031    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 02:58:09.166391 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:58:09.166354    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:58:11.993786 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:11.993754    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"]
Oct 30 02:58:11.994454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:11.994439    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5b3018a4-a1be-4bc3-a528-c0e27a2329b2" containerName="revoke-keys"
Oct 30 02:58:11.994506 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:11.994455    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3018a4-a1be-4bc3-a528-c0e27a2329b2" containerName="revoke-keys"
Oct 30 02:58:11.994556 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:11.994547    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3018a4-a1be-4bc3-a528-c0e27a2329b2" containerName="revoke-keys"
Oct 30 02:58:11.998280 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:11.998263    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.018190 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.018161    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"]
Oct 30 02:58:12.023221 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.023197    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:12.023410 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.023292    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:12.034743 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.034701    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.034934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.034915    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.035034 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.034950    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8482\" (UniqueName: \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.136175 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.136139    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.136414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.136257    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.136414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.136274    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8482\" (UniqueName: \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.136905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.136884    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.138544 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.138516    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.156501 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.156472    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8482\" (UniqueName: \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\") pod \"maas-api-revoke-sub-1d0786f0-5rftw\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.315954 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.315925    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:12.630454 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:12.630423    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85c827b_3ea4_41d0_8d57_83b658e9d836.slice/crio-2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189 WatchSource:0}: Error finding container 2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189: Status 404 returned error can't find the container with id 2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189
Oct 30 02:58:12.631366 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.631344    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"]
Oct 30 02:58:12.969137 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.969102    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw" event={"ID":"e85c827b-3ea4-41d0-8d57-83b658e9d836","Type":"ContainerStarted","Data":"a6b97f194376faeff78086408720046ce932c689c30c80a2950e2c01089da230"}
Oct 30 02:58:12.969309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:12.969162    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw" event={"ID":"e85c827b-3ea4-41d0-8d57-83b658e9d836","Type":"ContainerStarted","Data":"2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189"}
Oct 30 02:58:13.013042 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:13.012985    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw" podStartSLOduration=2.012970794 podStartE2EDuration="2.012970794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:13.009867082 +0000 UTC m=+1032.084037878" watchObservedRunningTime="2026-10-30 02:58:13.012970794 +0000 UTC m=+1032.087141587"
Oct 30 02:58:13.978952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:13.978837    4214 generic.go:356] "Generic (PLEG): container finished" podID="e85c827b-3ea4-41d0-8d57-83b658e9d836" containerID="a6b97f194376faeff78086408720046ce932c689c30c80a2950e2c01089da230" exitCode=0
Oct 30 02:58:13.978952 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:13.978918    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw" event={"ID":"e85c827b-3ea4-41d0-8d57-83b658e9d836","Type":"ContainerDied","Data":"a6b97f194376faeff78086408720046ce932c689c30c80a2950e2c01089da230"}
Oct 30 02:58:15.174078 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.173949    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:15.319520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.319487    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\") pod \"e85c827b-3ea4-41d0-8d57-83b658e9d836\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") "
Oct 30 02:58:15.319716 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.319569    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\") pod \"e85c827b-3ea4-41d0-8d57-83b658e9d836\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") "
Oct 30 02:58:15.319803 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.319783    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\" (UniqueName: \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\") pod \"e85c827b-3ea4-41d0-8d57-83b658e9d836\" (UID: \"e85c827b-3ea4-41d0-8d57-83b658e9d836\") "
Oct 30 02:58:15.319913 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.319892    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca" pod "e85c827b-3ea4-41d0-8d57-83b658e9d836" (UID: "e85c827b-3ea4-41d0-8d57-83b658e9d836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:15.320315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.320295    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:15.321620 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.321597    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca" pod "e85c827b-3ea4-41d0-8d57-83b658e9d836" (UID: "e85c827b-3ea4-41d0-8d57-83b658e9d836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:15.321731 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.321716    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482" pod "e85c827b-3ea4-41d0-8d57-83b658e9d836" (UID: "e85c827b-3ea4-41d0-8d57-83b658e9d836"). InnerVolumeSpecName "kube-api-access-v8482". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:15.422200 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.422165    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8482\" (UniqueName: \"kubernetes.io/projected/e85c827b-3ea4-41d0-8d57-83b658e9d836-kube-api-access-v8482\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:15.422514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.422493    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e85c827b-3ea4-41d0-8d57-83b658e9d836-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:15.999277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.999248    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"
Oct 30 02:58:15.999277 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.999267    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw" event={"ID":"e85c827b-3ea4-41d0-8d57-83b658e9d836","Type":"ContainerDied","Data":"2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189"}
Oct 30 02:58:15.999523 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:15.999316    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b73883d7312a8a4e8473d47630fd68c5dd4d08c36878ceb1258649d0a045189"
Oct 30 02:58:17.120625 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.120595    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"]
Oct 30 02:58:17.121291 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.121274    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e85c827b-3ea4-41d0-8d57-83b658e9d836" containerName="revoke-keys"
Oct 30 02:58:17.121291 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.121288    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85c827b-3ea4-41d0-8d57-83b658e9d836" containerName="revoke-keys"
Oct 30 02:58:17.121426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.121384    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85c827b-3ea4-41d0-8d57-83b658e9d836" containerName="revoke-keys"
Oct 30 02:58:17.125214 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.125197    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.158253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.145035    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"]
Oct 30 02:58:17.158253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.150370    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:17.158253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.151098    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:17.282258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.282194    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.282458 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.282337    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.289251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.282524    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkg2\" (UniqueName: \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.383859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.383773    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkg2\" (UniqueName: \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.384036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.383875    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.384036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.383932    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.384635 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.384605    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.386166 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.386145    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.404611 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.404589    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkg2\" (UniqueName: \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\") pod \"maas-api-revoke-sub-03f31944-4gcns\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.440285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.440262    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:17.751528 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:17.751494    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"]
Oct 30 02:58:18.018022 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:18.017945    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns" event={"ID":"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b","Type":"ContainerStarted","Data":"d70c387576796563520cb956d381abc7c03fa418e93d9e92f8d55892c30d0ed3"}
Oct 30 02:58:19.027962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:19.027874    4214 generic.go:356] "Generic (PLEG): container finished" podID="fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" containerID="26abb4738da54ecae30d485693e8140720eb6f46f181449a2ee7a06e6d5b19d3" exitCode=0
Oct 30 02:58:19.028362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:19.027955    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns" event={"ID":"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b","Type":"ContainerDied","Data":"26abb4738da54ecae30d485693e8140720eb6f46f181449a2ee7a06e6d5b19d3"}
Oct 30 02:58:21.588003 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:21.504810    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.706275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:21.692372    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.001821 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.001745    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:22.156012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.155972    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\" (UniqueName: \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\") pod \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") "
Oct 30 02:58:22.156392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.156362    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\") pod \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") "
Oct 30 02:58:22.178563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.178514    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\") pod \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\" (UID: \"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b\") "
Oct 30 02:58:22.213278 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.200912    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca" pod "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" (UID: "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:22.286846 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.283989    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.309260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.297889    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:22.309582 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.309530    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.457665 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.438673    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2" pod "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" (UID: "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b"). InnerVolumeSpecName "kube-api-access-trkg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:22.499123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.499084    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca" pod "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" (UID: "fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:22.583818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.524519    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkg2\" (UniqueName: \"kubernetes.io/projected/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-kube-api-access-trkg2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:22.583818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.524560    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:22.823171 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.823140    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"
Oct 30 02:58:22.845525 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.845427    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns" event={"ID":"fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b","Type":"ContainerDied","Data":"d70c387576796563520cb956d381abc7c03fa418e93d9e92f8d55892c30d0ed3"}
Oct 30 02:58:22.880746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.845906    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70c387576796563520cb956d381abc7c03fa418e93d9e92f8d55892c30d0ed3"
Oct 30 02:58:22.880967 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.855374    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.899264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.875550    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.918254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:22.900411    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:58:23.334255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.333579    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.359254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.338476    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.359254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.347158    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.359254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.347224    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.485559 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.485515    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:58:23.485902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.485806    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:58:23.495363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.495326    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.495574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.495552    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.757045 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.756961    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.757331 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.757296    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.923421 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.917354    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.945274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.936370    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.945274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:23.945068    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:24.205861 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.205804    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:24.206176 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.206126    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.363312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.350901    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.409419 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.409377    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:24.409700 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.409669    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.482308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.481673    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.756161 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.615375    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:24.756161 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.615441    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.939511 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.939460    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.965263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:24.954501    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.082412 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:25.047407    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt" podUID="74cb6b35-cc28-4b86-be20-3d994dd660bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.432422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:25.429073    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:25.489031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:25.488970    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.492946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:25.492909    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:26.000270 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:25.994391    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:26.088248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.078268    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:26.128602 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:58:26.128565    4214 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.069s"
Oct 30 02:58:26.430365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.430315    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:26.493368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.493316    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:26.619411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.619360    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:26.619747 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.619715    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:26.840934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.826284    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"]
Oct 30 02:58:26.840934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.827301    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" containerName="revoke-keys"
Oct 30 02:58:26.840934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.827322    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" containerName="revoke-keys"
Oct 30 02:58:26.840934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.827518    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" containerName="revoke-keys"
Oct 30 02:58:26.840934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.832194    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:26.922253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.899645    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:26.922253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:26.899730    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:27.015829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.015792    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"]
Oct 30 02:58:27.104308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.062083    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.104308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.062330    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.104308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.062369    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455nm\" (UniqueName: \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.104308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.096358    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:27.235675 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.016508    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:27.347254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.345199    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.347254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.345654    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.347254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.345695    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455nm\" (UniqueName: \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.380301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.379731    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.452992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.452946    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 02:58:27.480390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.466536    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.480390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.466638    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:27.480390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.475819    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455nm\" (UniqueName: \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\") pod \"maas-api-revoke-sub-3b6809c4-p99wp\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:27.496889 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.496849    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:27.593296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:27.592465    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:28.162253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.096739    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:28.300477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.241173    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:28.475484 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.475378    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 02:58:28.476029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.476002    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:28.512816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.499325    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:28.571876 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.571852    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:28.581762 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.581734    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:58:28.959252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.940802    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:58:28.988910 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:28.975610    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:58:29.294638 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:29.294605    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc25c04_cb04_4ca3_a7a1_466b98bf9763.slice/crio-b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5 WatchSource:0}: Error finding container b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5: Status 404 returned error can't find the container with id b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5
Oct 30 02:58:29.302783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:29.302756    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"]
Oct 30 02:58:29.625032 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:29.624993    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp" event={"ID":"8cc25c04-cb04-4ca3-a7a1-466b98bf9763","Type":"ContainerStarted","Data":"b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5"}
Oct 30 02:58:30.634158 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:30.634119    4214 generic.go:356] "Generic (PLEG): container finished" podID="8cc25c04-cb04-4ca3-a7a1-466b98bf9763" containerID="9c3547a96e7dc84f3be52ec599c46214e39519cbcaebf3a86a20733837aad017" exitCode=0
Oct 30 02:58:30.634640 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:30.634173    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp" event={"ID":"8cc25c04-cb04-4ca3-a7a1-466b98bf9763","Type":"ContainerDied","Data":"9c3547a96e7dc84f3be52ec599c46214e39519cbcaebf3a86a20733837aad017"}
Oct 30 02:58:31.812730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.812706    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:31.889865 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.889834    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\") pod \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") "
Oct 30 02:58:31.890107 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.890089    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\") pod \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") "
Oct 30 02:58:31.890254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.890212    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\" (UniqueName: \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\") pod \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\" (UID: \"8cc25c04-cb04-4ca3-a7a1-466b98bf9763\") "
Oct 30 02:58:31.890581 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.890488    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca" pod "8cc25c04-cb04-4ca3-a7a1-466b98bf9763" (UID: "8cc25c04-cb04-4ca3-a7a1-466b98bf9763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:31.890924 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.890900    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:31.892350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.892289    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm" pod "8cc25c04-cb04-4ca3-a7a1-466b98bf9763" (UID: "8cc25c04-cb04-4ca3-a7a1-466b98bf9763"). InnerVolumeSpecName "kube-api-access-455nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:31.892451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.892436    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca" pod "8cc25c04-cb04-4ca3-a7a1-466b98bf9763" (UID: "8cc25c04-cb04-4ca3-a7a1-466b98bf9763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:31.992730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.992704    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455nm\" (UniqueName: \"kubernetes.io/projected/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-kube-api-access-455nm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:31.992730 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:31.992730    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc25c04-cb04-4ca3-a7a1-466b98bf9763-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:32.703839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:32.694855    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp" event={"ID":"8cc25c04-cb04-4ca3-a7a1-466b98bf9763","Type":"ContainerDied","Data":"b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5"}
Oct 30 02:58:32.703839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:32.694959    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b44558fb2c8c0caed1c01ee2d60996aca17c881b1c451d1ce304b1c27319c9e5"
Oct 30 02:58:32.703839 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:32.695070    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"
Oct 30 02:58:33.145099 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.145070    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"]
Oct 30 02:58:33.145962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.145944    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8cc25c04-cb04-4ca3-a7a1-466b98bf9763" containerName="revoke-keys"
Oct 30 02:58:33.146036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.145964    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc25c04-cb04-4ca3-a7a1-466b98bf9763" containerName="revoke-keys"
Oct 30 02:58:33.146085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.146067    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc25c04-cb04-4ca3-a7a1-466b98bf9763" containerName="revoke-keys"
Oct 30 02:58:33.149905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.149886    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.166992 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.166958    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"]
Oct 30 02:58:33.173961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.173940    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:33.174131 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.174109    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:33.223807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.223773    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v85n\" (UniqueName: \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.223955 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.223911    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.224012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.223976    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.325141 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.325112    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.325364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.325340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v85n\" (UniqueName: \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.325497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.325475    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.325936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.325901    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.327583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.327559    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.344881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.344861    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v85n\" (UniqueName: \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\") pod \"maas-api-revoke-sub-5c9e20c6-mzcpc\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.465099 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.465028    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:33.771565 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:33.771523    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76b8c43_53fb_43a1_93dc_8be38e7e3062.slice/crio-e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467 WatchSource:0}: Error finding container e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467: Status 404 returned error can't find the container with id e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467
Oct 30 02:58:33.773149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:33.773122    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"]
Oct 30 02:58:34.713752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:34.713722    4214 generic.go:356] "Generic (PLEG): container finished" podID="e76b8c43-53fb-43a1-93dc-8be38e7e3062" containerID="e04096487540cc12695d5eaaa1aeee82acea0f94937ff395ad2d77553e242724" exitCode=0
Oct 30 02:58:34.714072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:34.713775    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc" event={"ID":"e76b8c43-53fb-43a1-93dc-8be38e7e3062","Type":"ContainerDied","Data":"e04096487540cc12695d5eaaa1aeee82acea0f94937ff395ad2d77553e242724"}
Oct 30 02:58:34.714072 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:34.713832    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc" event={"ID":"e76b8c43-53fb-43a1-93dc-8be38e7e3062","Type":"ContainerStarted","Data":"e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467"}
Oct 30 02:58:35.989831 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:35.989760    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:36.070452 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.070415    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\" (UniqueName: \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\") pod \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") "
Oct 30 02:58:36.070622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.070551    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\") pod \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") "
Oct 30 02:58:36.070622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.070611    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\") pod \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\" (UID: \"e76b8c43-53fb-43a1-93dc-8be38e7e3062\") "
Oct 30 02:58:36.070962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.070925    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca" pod "e76b8c43-53fb-43a1-93dc-8be38e7e3062" (UID: "e76b8c43-53fb-43a1-93dc-8be38e7e3062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:36.071204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.071182    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:36.073012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.072990    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n" pod "e76b8c43-53fb-43a1-93dc-8be38e7e3062" (UID: "e76b8c43-53fb-43a1-93dc-8be38e7e3062"). InnerVolumeSpecName "kube-api-access-8v85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:36.073498 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.073467    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca" pod "e76b8c43-53fb-43a1-93dc-8be38e7e3062" (UID: "e76b8c43-53fb-43a1-93dc-8be38e7e3062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:36.172574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.172530    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76b8c43-53fb-43a1-93dc-8be38e7e3062-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:36.172574 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.172570    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v85n\" (UniqueName: \"kubernetes.io/projected/e76b8c43-53fb-43a1-93dc-8be38e7e3062-kube-api-access-8v85n\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:36.732422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.732396    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"
Oct 30 02:58:36.732567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.732434    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc" event={"ID":"e76b8c43-53fb-43a1-93dc-8be38e7e3062","Type":"ContainerDied","Data":"e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467"}
Oct 30 02:58:36.732567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:36.732489    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e110e766f4f5bb2260c13f84c4cebfe78bb9d5734e88ad1e56b8df92ccd6c467"
Oct 30 02:58:39.169347 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:58:39.169305    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:58:43.445601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.445568    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"]
Oct 30 02:58:43.446253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.446224    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e76b8c43-53fb-43a1-93dc-8be38e7e3062" containerName="revoke-keys"
Oct 30 02:58:43.446253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.446252    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76b8c43-53fb-43a1-93dc-8be38e7e3062" containerName="revoke-keys"
Oct 30 02:58:43.446390 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.446367    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76b8c43-53fb-43a1-93dc-8be38e7e3062" containerName="revoke-keys"
Oct 30 02:58:43.449918 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.449897    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.468055 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.468026    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"]
Oct 30 02:58:43.474225 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.474196    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:43.474423 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.474214    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:43.604165 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.604127    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dl7\" (UniqueName: \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.604397 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.604178    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.604521 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.604499    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.715751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.714269    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.715751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.714363    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dl7\" (UniqueName: \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.715751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.714400    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.715751 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.715380    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.730213 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.726075    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.767565 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.764521    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dl7\" (UniqueName: \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\") pod \"maas-api-revoke-sub-f8d358ab-sw6d9\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:43.771268 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:43.768742    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:44.118576 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:44.118546    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec176c05_9c6b_4e24_9ac2_d3a4c586b72e.slice/crio-3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409 WatchSource:0}: Error finding container 3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409: Status 404 returned error can't find the container with id 3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409
Oct 30 02:58:44.120116 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:44.120084    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"]
Oct 30 02:58:44.829962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:44.829920    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9" event={"ID":"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e","Type":"ContainerStarted","Data":"278c87c4328eefb0af54b8529f49b7545dcf5dea592dbedde9a965e4cf36f646"}
Oct 30 02:58:44.830452 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:44.829993    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9" event={"ID":"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e","Type":"ContainerStarted","Data":"3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409"}
Oct 30 02:58:44.874001 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:44.873949    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9" podStartSLOduration=1.8739370370000001 podStartE2EDuration="1.873937037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:44.870010015 +0000 UTC m=+1063.944180819" watchObservedRunningTime="2026-10-30 02:58:44.873937037 +0000 UTC m=+1063.948107830"
Oct 30 02:58:45.841122 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:45.841088    4214 generic.go:356] "Generic (PLEG): container finished" podID="ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" containerID="278c87c4328eefb0af54b8529f49b7545dcf5dea592dbedde9a965e4cf36f646" exitCode=0
Oct 30 02:58:45.841529 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:45.841168    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9" event={"ID":"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e","Type":"ContainerDied","Data":"278c87c4328eefb0af54b8529f49b7545dcf5dea592dbedde9a965e4cf36f646"}
Oct 30 02:58:47.060391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.060362    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:47.082645 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.082617    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\") pod \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") "
Oct 30 02:58:47.082798 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.082687    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\" (UniqueName: \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\") pod \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") "
Oct 30 02:58:47.082884 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.082862    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca" pod "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" (UID: "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:47.082958 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.082887    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\") pod \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\" (UID: \"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e\") "
Oct 30 02:58:47.083813 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.083784    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:47.085118 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.085092    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7" pod "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" (UID: "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e"). InnerVolumeSpecName "kube-api-access-d2dl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:47.085338 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.085316    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca" pod "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" (UID: "ec176c05-9c6b-4e24-9ac2-d3a4c586b72e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:47.227300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.219350    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:47.227300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:47.219379    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2dl7\" (UniqueName: \"kubernetes.io/projected/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e-kube-api-access-d2dl7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:58:48.089944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.089891    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9" event={"ID":"ec176c05-9c6b-4e24-9ac2-d3a4c586b72e","Type":"ContainerDied","Data":"3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409"}
Oct 30 02:58:48.095252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.090355    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"
Oct 30 02:58:48.149486 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.149421    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eacdf5f1165d9c6b96b9a4bac7f416597c203542f24b96bcd4ac9a71eed7409"
Oct 30 02:58:48.415427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.415326    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:48.438379 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.438329    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:48.500382 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.500329    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:48.977016 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:48.975299    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.091253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:49.087343    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.447256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:49.443322    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.447256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:49.443776    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:58:49.702929 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:49.702827    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.011530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:49.820112    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:58:50.029323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.012968    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.029323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.016023    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:58:50.161191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.161130    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.343273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.343188    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:50.585257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.504467    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.713486 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:50.713341    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.043311 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.020092    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.323313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.293838    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:51.659128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.659008    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.718216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.717407    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.806563 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.799199    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.829364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.829156    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:51.829364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:51.829224    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.325274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:52.304881    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.325274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:52.306742    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.723932 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:52.723831    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.820382 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:52.820316    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:53.331252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.327075    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.366362    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.366716    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.366808    4214 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.398714    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.398823    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.399168    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.399285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.399204    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.516746    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.516815    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.533341    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.533406    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.533519    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.535109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.533548    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.563475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.563438    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.563742 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.563713    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.738114 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.738010    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.781977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.781842    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:53.781977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.781906    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:53.844792 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.843483    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:53.893510 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.893470    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"]
Oct 30 02:58:53.904269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:53.895650    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" containerName="revoke-keys"
Oct 30 02:58:54.078501 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.078470    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" containerName="revoke-keys"
Oct 30 02:58:54.079084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.079068    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" containerName="revoke-keys"
Oct 30 02:58:54.343963 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.343850    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:54.344598 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.344559    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:54.371606 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.371560    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:54.390297 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.372582    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:54.690528 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.690441    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"]
Oct 30 02:58:54.709257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.704406    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.735253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.725829    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:58:54.735253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.725914    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:58:54.746204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.746052    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:54.795621 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.795574    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gmz\" (UniqueName: \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.796997 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.796968    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.853356 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.853315    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.875069 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.875020    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:58:54.968303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.959613    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.968303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.959687    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.968303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.959746    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gmz\" (UniqueName: \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.971042 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.971009    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:54.993577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:54.993543    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:55.029026 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.028992    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gmz\" (UniqueName: \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\") pod \"maas-api-revoke-sub-bfe0cffc-2xzg6\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:55.136254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.135738    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:58:55.307837 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.307426    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 02:58:55.374293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.373474    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:55.374293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.373858    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:58:55.765394 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.764342    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:58:55.862178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:55.862146    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:58:56.375286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.374369    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:56.376627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.375869    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:56.421345 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.419461    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:58:56.518622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.518588    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:58:56.533957 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.533933    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:58:56.621677 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.613649    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"]
Oct 30 02:58:56.621677 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:58:56.616502    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43afc460_e3bd_4527_aade_e879b6b02695.slice/crio-6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512 WatchSource:0}: Error finding container 6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512: Status 404 returned error can't find the container with id 6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512
Oct 30 02:58:56.825827 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:56.825775    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6" event={"ID":"43afc460-e3bd-4527-aade-e879b6b02695","Type":"ContainerStarted","Data":"6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512"}
Oct 30 02:58:57.844492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:57.844448    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6" event={"ID":"43afc460-e3bd-4527-aade-e879b6b02695","Type":"ContainerStarted","Data":"4f832b6729c2bc0ee4a9b3683cfb4da67d4ae407dad8f0d661c1bb6dc6614f6c"}
Oct 30 02:58:57.890306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:57.890253    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6" podStartSLOduration=4.890240246 podStartE2EDuration="4.890240246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:57.885316041 +0000 UTC m=+1076.959486833" watchObservedRunningTime="2026-10-30 02:58:57.890240246 +0000 UTC m=+1076.964411033"
Oct 30 02:58:58.861566 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:58.861529    4214 generic.go:356] "Generic (PLEG): container finished" podID="43afc460-e3bd-4527-aade-e879b6b02695" containerID="4f832b6729c2bc0ee4a9b3683cfb4da67d4ae407dad8f0d661c1bb6dc6614f6c" exitCode=0
Oct 30 02:58:58.862111 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:58:58.861608    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6" event={"ID":"43afc460-e3bd-4527-aade-e879b6b02695","Type":"ContainerDied","Data":"4f832b6729c2bc0ee4a9b3683cfb4da67d4ae407dad8f0d661c1bb6dc6614f6c"}
Oct 30 02:59:00.038801 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.038774    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:59:00.165643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.165613    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\" (UniqueName: \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\") pod \"43afc460-e3bd-4527-aade-e879b6b02695\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") "
Oct 30 02:59:00.165810 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.165662    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\") pod \"43afc460-e3bd-4527-aade-e879b6b02695\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") "
Oct 30 02:59:00.165874 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.165830    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\") pod \"43afc460-e3bd-4527-aade-e879b6b02695\" (UID: \"43afc460-e3bd-4527-aade-e879b6b02695\") "
Oct 30 02:59:00.166038 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.166014    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca" pod "43afc460-e3bd-4527-aade-e879b6b02695" (UID: "43afc460-e3bd-4527-aade-e879b6b02695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:00.166520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.166494    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:00.167805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.167774    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca" pod "43afc460-e3bd-4527-aade-e879b6b02695" (UID: "43afc460-e3bd-4527-aade-e879b6b02695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:00.167902 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.167884    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz" pod "43afc460-e3bd-4527-aade-e879b6b02695" (UID: "43afc460-e3bd-4527-aade-e879b6b02695"). InnerVolumeSpecName "kube-api-access-r2gmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:00.267797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.267706    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43afc460-e3bd-4527-aade-e879b6b02695-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:00.267797 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.267736    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2gmz\" (UniqueName: \"kubernetes.io/projected/43afc460-e3bd-4527-aade-e879b6b02695-kube-api-access-r2gmz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:00.712260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.712209    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"]
Oct 30 02:59:00.712881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.712866    4214 cpu_manager.go:414] "containerMap: removing container" podUID="43afc460-e3bd-4527-aade-e879b6b02695" containerName="revoke-keys"
Oct 30 02:59:00.712940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.712882    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43afc460-e3bd-4527-aade-e879b6b02695" containerName="revoke-keys"
Oct 30 02:59:00.713011 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.713002    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="43afc460-e3bd-4527-aade-e879b6b02695" containerName="revoke-keys"
Oct 30 02:59:00.716776 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.716761    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.733889 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.733863    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"]
Oct 30 02:59:00.774882 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.774848    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.775147 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.775124    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cj8\" (UniqueName: \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.775309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.775293    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.876223 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.876186    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cj8\" (UniqueName: \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.876424 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.876341    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.876424 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.876404    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.876983 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.876962    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.878842 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.878821    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:00.881816 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.881795    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"
Oct 30 02:59:00.881895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.881791    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6" event={"ID":"43afc460-e3bd-4527-aade-e879b6b02695","Type":"ContainerDied","Data":"6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512"}
Oct 30 02:59:00.881944 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.881924    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c03aa016d7c6a88f87367cb855fc8415b4d1042380e1067a7fae30f8a8e0512"
Oct 30 02:59:00.896938 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:00.896918    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cj8\" (UniqueName: \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\") pod \"maas-api-revoke-sub-9e2015db-c64kb\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:01.032733 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.032665    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:01.349090 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:01.349053    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a383e0_4b39_4356_a4fa_2678a8f5de0a.slice/crio-c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e WatchSource:0}: Error finding container c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e: Status 404 returned error can't find the container with id c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e
Oct 30 02:59:01.352091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.352060    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"]
Oct 30 02:59:01.482280 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.482248    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"]
Oct 30 02:59:01.488301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.488278    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.504445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.504418    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"]
Oct 30 02:59:01.592603 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.592567    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qfh\" (UniqueName: \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.592802 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.592637    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.592870 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.592820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.694321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.694279    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qfh\" (UniqueName: \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.694513 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.694360    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.694650 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.694632    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.695296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.695275    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.696689 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.696670    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.714358 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.714328    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qfh\" (UniqueName: \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\") pod \"maas-api-revoke-sub-743c3e8c-zrjsn\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.810849 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.810823    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:01.893993 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.893958    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb" event={"ID":"23a383e0-4b39-4356-a4fa-2678a8f5de0a","Type":"ContainerStarted","Data":"62d6b75d908a41974a2a3e0fe1cceaf95464318eae922e395733cde0bde0471f"}
Oct 30 02:59:01.894119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.894023    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb" event={"ID":"23a383e0-4b39-4356-a4fa-2678a8f5de0a","Type":"ContainerStarted","Data":"c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e"}
Oct 30 02:59:01.939752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:01.939695    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb" podStartSLOduration=1.9396761630000001 podStartE2EDuration="1.939676163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:01.932749492 +0000 UTC m=+1081.006920290" watchObservedRunningTime="2026-10-30 02:59:01.939676163 +0000 UTC m=+1081.013846955"
Oct 30 02:59:02.127655 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:02.127624    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1cc68a_0765_4bec_a2b0_623de738cfef.slice/crio-ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56 WatchSource:0}: Error finding container ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56: Status 404 returned error can't find the container with id ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56
Oct 30 02:59:02.128079 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.128057    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"]
Oct 30 02:59:02.904896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.904860    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn" event={"ID":"0d1cc68a-0765-4bec-a2b0-623de738cfef","Type":"ContainerStarted","Data":"555a256a193b4371875a47acbdfa60c71f9a9a45d503e41a6efb410f303e1ea5"}
Oct 30 02:59:02.905425 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.904929    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn" event={"ID":"0d1cc68a-0765-4bec-a2b0-623de738cfef","Type":"ContainerStarted","Data":"ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56"}
Oct 30 02:59:02.906668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.906640    4214 generic.go:356] "Generic (PLEG): container finished" podID="23a383e0-4b39-4356-a4fa-2678a8f5de0a" containerID="62d6b75d908a41974a2a3e0fe1cceaf95464318eae922e395733cde0bde0471f" exitCode=0
Oct 30 02:59:02.906807 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.906720    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb" event={"ID":"23a383e0-4b39-4356-a4fa-2678a8f5de0a","Type":"ContainerDied","Data":"62d6b75d908a41974a2a3e0fe1cceaf95464318eae922e395733cde0bde0471f"}
Oct 30 02:59:02.950790 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:02.950733    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn" podStartSLOduration=1.950716023 podStartE2EDuration="1.950716023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:02.943112148 +0000 UTC m=+1082.017282941" watchObservedRunningTime="2026-10-30 02:59:02.950716023 +0000 UTC m=+1082.024886817"
Oct 30 02:59:03.921152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:03.921114    4214 generic.go:356] "Generic (PLEG): container finished" podID="0d1cc68a-0765-4bec-a2b0-623de738cfef" containerID="555a256a193b4371875a47acbdfa60c71f9a9a45d503e41a6efb410f303e1ea5" exitCode=0
Oct 30 02:59:03.921623 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:03.921197    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn" event={"ID":"0d1cc68a-0765-4bec-a2b0-623de738cfef","Type":"ContainerDied","Data":"555a256a193b4371875a47acbdfa60c71f9a9a45d503e41a6efb410f303e1ea5"}
Oct 30 02:59:04.098173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.098146    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:04.136905 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.136873    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\") pod \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") "
Oct 30 02:59:04.137066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.137028    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\" (UniqueName: \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\") pod \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") "
Oct 30 02:59:04.137153 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.137126    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\") pod \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\" (UID: \"23a383e0-4b39-4356-a4fa-2678a8f5de0a\") "
Oct 30 02:59:04.137514 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.137487    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca" pod "23a383e0-4b39-4356-a4fa-2678a8f5de0a" (UID: "23a383e0-4b39-4356-a4fa-2678a8f5de0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:04.140757 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.140732    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:04.142023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.141981    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8" pod "23a383e0-4b39-4356-a4fa-2678a8f5de0a" (UID: "23a383e0-4b39-4356-a4fa-2678a8f5de0a"). InnerVolumeSpecName "kube-api-access-b6cj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:04.142202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.142176    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca" pod "23a383e0-4b39-4356-a4fa-2678a8f5de0a" (UID: "23a383e0-4b39-4356-a4fa-2678a8f5de0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:04.242483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.242414    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6cj8\" (UniqueName: \"kubernetes.io/projected/23a383e0-4b39-4356-a4fa-2678a8f5de0a-kube-api-access-b6cj8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:04.242483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.242441    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a383e0-4b39-4356-a4fa-2678a8f5de0a-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:04.932435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.932408    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"
Oct 30 02:59:04.932844 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.932442    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb" event={"ID":"23a383e0-4b39-4356-a4fa-2678a8f5de0a","Type":"ContainerDied","Data":"c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e"}
Oct 30 02:59:04.932844 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:04.932499    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e74ccd97e8125df2faacb9c3a645de33a38bdd3e648a67f16469be8438d43e"
Oct 30 02:59:05.097427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.097404    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:05.157920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.157882    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\") pod \"0d1cc68a-0765-4bec-a2b0-623de738cfef\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") "
Oct 30 02:59:05.158084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.158044    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\") pod \"0d1cc68a-0765-4bec-a2b0-623de738cfef\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") "
Oct 30 02:59:05.158149 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.158129    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\" (UniqueName: \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\") pod \"0d1cc68a-0765-4bec-a2b0-623de738cfef\" (UID: \"0d1cc68a-0765-4bec-a2b0-623de738cfef\") "
Oct 30 02:59:05.158493 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.158282    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca" pod "0d1cc68a-0765-4bec-a2b0-623de738cfef" (UID: "0d1cc68a-0765-4bec-a2b0-623de738cfef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:05.158946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.158922    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:05.160661 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.160633    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh" pod "0d1cc68a-0765-4bec-a2b0-623de738cfef" (UID: "0d1cc68a-0765-4bec-a2b0-623de738cfef"). InnerVolumeSpecName "kube-api-access-v4qfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:05.160782 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.160758    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca" pod "0d1cc68a-0765-4bec-a2b0-623de738cfef" (UID: "0d1cc68a-0765-4bec-a2b0-623de738cfef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:05.259976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.259893    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d1cc68a-0765-4bec-a2b0-623de738cfef-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:05.259976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.259925    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qfh\" (UniqueName: \"kubernetes.io/projected/0d1cc68a-0765-4bec-a2b0-623de738cfef-kube-api-access-v4qfh\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:05.629068 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629035    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"]
Oct 30 02:59:05.629733 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629718    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0d1cc68a-0765-4bec-a2b0-623de738cfef" containerName="revoke-keys"
Oct 30 02:59:05.629776 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629736    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1cc68a-0765-4bec-a2b0-623de738cfef" containerName="revoke-keys"
Oct 30 02:59:05.629776 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629772    4214 cpu_manager.go:414] "containerMap: removing container" podUID="23a383e0-4b39-4356-a4fa-2678a8f5de0a" containerName="revoke-keys"
Oct 30 02:59:05.629886 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629778    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a383e0-4b39-4356-a4fa-2678a8f5de0a" containerName="revoke-keys"
Oct 30 02:59:05.629886 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629879    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1cc68a-0765-4bec-a2b0-623de738cfef" containerName="revoke-keys"
Oct 30 02:59:05.629946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.629895    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a383e0-4b39-4356-a4fa-2678a8f5de0a" containerName="revoke-keys"
Oct 30 02:59:05.633705 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.633684    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.655728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.655696    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"]
Oct 30 02:59:05.666517 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.666483    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncr4\" (UniqueName: \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.666752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.666725    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.666881 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.666829    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.768411 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.768376    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.768579 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.768462    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.768718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.768693    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncr4\" (UniqueName: \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.769134 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.769114    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.770734 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.770707    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.788809 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.788776    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncr4\" (UniqueName: \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\") pod \"maas-api-revoke-sub-34292810-vrt8c\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:05.943895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.943809    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn" event={"ID":"0d1cc68a-0765-4bec-a2b0-623de738cfef","Type":"ContainerDied","Data":"ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56"}
Oct 30 02:59:05.943895 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.943871    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7c99d45e792e09f0b2e69062651a2a98f8a746fab8ae00afa9a31b25bf8b56"
Oct 30 02:59:05.944323 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.943891    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"
Oct 30 02:59:05.957116 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:05.957091    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:06.281736 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:06.281705    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"]
Oct 30 02:59:06.954001 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:06.953919    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c" event={"ID":"562212ed-d68e-48af-ad4c-e784e3c07859","Type":"ContainerStarted","Data":"5bd36502db3ecc308f4b2d4312773a8d9959c2ce1536a1a8896d3bd0ba676a8a"}
Oct 30 02:59:06.954001 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:06.953987    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c" event={"ID":"562212ed-d68e-48af-ad4c-e784e3c07859","Type":"ContainerStarted","Data":"ec86ee870d141866f47a4c860d4f7cfe159a3b2109290dba3aeb80df4f8df2db"}
Oct 30 02:59:06.997884 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:06.997838    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c" podStartSLOduration=1.997822939 podStartE2EDuration="1.997822939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:06.993059008 +0000 UTC m=+1086.067229801" watchObservedRunningTime="2026-10-30 02:59:06.997822939 +0000 UTC m=+1086.071993750"
Oct 30 02:59:07.965763 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:07.965727    4214 generic.go:356] "Generic (PLEG): container finished" podID="562212ed-d68e-48af-ad4c-e784e3c07859" containerID="5bd36502db3ecc308f4b2d4312773a8d9959c2ce1536a1a8896d3bd0ba676a8a" exitCode=0
Oct 30 02:59:07.966168 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:07.965809    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c" event={"ID":"562212ed-d68e-48af-ad4c-e784e3c07859","Type":"ContainerDied","Data":"5bd36502db3ecc308f4b2d4312773a8d9959c2ce1536a1a8896d3bd0ba676a8a"}
Oct 30 02:59:09.158219 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.158190    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:09.168501 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:09.168465    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:59:09.261988 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.261906    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\") pod \"562212ed-d68e-48af-ad4c-e784e3c07859\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") "
Oct 30 02:59:09.262132 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.262061    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\" (UniqueName: \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\") pod \"562212ed-d68e-48af-ad4c-e784e3c07859\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") "
Oct 30 02:59:09.262305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.262283    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\") pod \"562212ed-d68e-48af-ad4c-e784e3c07859\" (UID: \"562212ed-d68e-48af-ad4c-e784e3c07859\") "
Oct 30 02:59:09.262401 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.262350    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca" pod "562212ed-d68e-48af-ad4c-e784e3c07859" (UID: "562212ed-d68e-48af-ad4c-e784e3c07859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:09.262946 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.262921    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:09.264519 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.264485    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4" pod "562212ed-d68e-48af-ad4c-e784e3c07859" (UID: "562212ed-d68e-48af-ad4c-e784e3c07859"). InnerVolumeSpecName "kube-api-access-vncr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:09.264962 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.264936    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca" pod "562212ed-d68e-48af-ad4c-e784e3c07859" (UID: "562212ed-d68e-48af-ad4c-e784e3c07859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:09.329421 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.329389    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"]
Oct 30 02:59:09.342830 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.342793    4214 cpu_manager.go:414] "containerMap: removing container" podUID="562212ed-d68e-48af-ad4c-e784e3c07859" containerName="revoke-keys"
Oct 30 02:59:09.343004 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.342848    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562212ed-d68e-48af-ad4c-e784e3c07859" containerName="revoke-keys"
Oct 30 02:59:09.343306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.343265    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="562212ed-d68e-48af-ad4c-e784e3c07859" containerName="revoke-keys"
Oct 30 02:59:09.351590 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.351559    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.365178 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.365141    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562212ed-d68e-48af-ad4c-e784e3c07859-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:09.365439 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.365420    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vncr4\" (UniqueName: \"kubernetes.io/projected/562212ed-d68e-48af-ad4c-e784e3c07859-kube-api-access-vncr4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:09.372654 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.372627    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"]
Oct 30 02:59:09.466284 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.466221    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.466470 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.466292    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhvx\" (UniqueName: \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.466530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.466473    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.568642 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.568597    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.568857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.568661    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhvx\" (UniqueName: \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.568857 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.568762    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.569607 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.569575    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.571049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.571019    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.588472 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.588450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhvx\" (UniqueName: \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\") pod \"maas-api-revoke-sub-c2556782-9smwd\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.701388 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.701359    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:09.999139 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.999051    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c" event={"ID":"562212ed-d68e-48af-ad4c-e784e3c07859","Type":"ContainerDied","Data":"ec86ee870d141866f47a4c860d4f7cfe159a3b2109290dba3aeb80df4f8df2db"}
Oct 30 02:59:09.999330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.999069    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"
Oct 30 02:59:09.999330 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:09.999178    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec86ee870d141866f47a4c860d4f7cfe159a3b2109290dba3aeb80df4f8df2db"
Oct 30 02:59:10.018109 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:10.018075    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"]
Oct 30 02:59:11.008556 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.008526    4214 generic.go:356] "Generic (PLEG): container finished" podID="09d4f09b-29dd-45e3-be9a-60d4771bc838" containerID="cff8032e3fa7e99cdaf6d831d71afa737f0bb9aee852ddba70d3a22d5e5e4391" exitCode=0
Oct 30 02:59:11.008936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.008650    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd" event={"ID":"09d4f09b-29dd-45e3-be9a-60d4771bc838","Type":"ContainerDied","Data":"cff8032e3fa7e99cdaf6d831d71afa737f0bb9aee852ddba70d3a22d5e5e4391"}
Oct 30 02:59:11.008936 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.008705    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd" event={"ID":"09d4f09b-29dd-45e3-be9a-60d4771bc838","Type":"ContainerStarted","Data":"96d2ad002e025e0a3dde771a65a07e4a6265514332b4684e1673645b561c8199"}
Oct 30 02:59:11.824293 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.824264    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"]
Oct 30 02:59:11.829496 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.829479    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:11.848934 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.848910    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"]
Oct 30 02:59:11.929648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.929618    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:11.929796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.929670    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2h8b\" (UniqueName: \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:11.929796 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:11.929780    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.030803 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.030773    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.031376 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.030862    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2h8b\" (UniqueName: \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.031376 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.030968    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.031552 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.031527    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.033543 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.033523    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.052258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.052200    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2h8b\" (UniqueName: \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\") pod \"maas-api-revoke-sub-05266894-5v9px\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.148506 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.148474    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:12.191897 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.191867    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:12.236728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.236365    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\") pod \"09d4f09b-29dd-45e3-be9a-60d4771bc838\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") "
Oct 30 02:59:12.236728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.236434    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\" (UniqueName: \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\") pod \"09d4f09b-29dd-45e3-be9a-60d4771bc838\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") "
Oct 30 02:59:12.236728 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.236555    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\") pod \"09d4f09b-29dd-45e3-be9a-60d4771bc838\" (UID: \"09d4f09b-29dd-45e3-be9a-60d4771bc838\") "
Oct 30 02:59:12.242272 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.241921    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca" pod "09d4f09b-29dd-45e3-be9a-60d4771bc838" (UID: "09d4f09b-29dd-45e3-be9a-60d4771bc838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:12.247492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.247453    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx" pod "09d4f09b-29dd-45e3-be9a-60d4771bc838" (UID: "09d4f09b-29dd-45e3-be9a-60d4771bc838"). InnerVolumeSpecName "kube-api-access-wnhvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:12.248146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.248104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca" pod "09d4f09b-29dd-45e3-be9a-60d4771bc838" (UID: "09d4f09b-29dd-45e3-be9a-60d4771bc838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:12.343217 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.343132    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:12.343217 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.343161    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4f09b-29dd-45e3-be9a-60d4771bc838-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:12.343217 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.343171    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhvx\" (UniqueName: \"kubernetes.io/projected/09d4f09b-29dd-45e3-be9a-60d4771bc838-kube-api-access-wnhvx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:12.459092 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:12.459063    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f072ce6_bff1_4b1c_bc0a_c23798419ebe.slice/crio-fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c WatchSource:0}: Error finding container fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c: Status 404 returned error can't find the container with id fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c
Oct 30 02:59:12.460113 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:12.460016    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"]
Oct 30 02:59:13.028718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.028624    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px" event={"ID":"7f072ce6-bff1-4b1c-bc0a-c23798419ebe","Type":"ContainerStarted","Data":"68cf81f186bc8b951d8f6d126e3a7c8b47a77da0475d22a2b44463272db4be51"}
Oct 30 02:59:13.028718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.028716    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px" event={"ID":"7f072ce6-bff1-4b1c-bc0a-c23798419ebe","Type":"ContainerStarted","Data":"fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c"}
Oct 30 02:59:13.030427 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.030407    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"
Oct 30 02:59:13.030891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.030440    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd" event={"ID":"09d4f09b-29dd-45e3-be9a-60d4771bc838","Type":"ContainerDied","Data":"96d2ad002e025e0a3dde771a65a07e4a6265514332b4684e1673645b561c8199"}
Oct 30 02:59:13.030891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.030476    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96d2ad002e025e0a3dde771a65a07e4a6265514332b4684e1673645b561c8199"
Oct 30 02:59:13.071128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.071077    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px" podStartSLOduration=2.071063732 podStartE2EDuration="2.071063732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:13.070566639 +0000 UTC m=+1092.144737431" watchObservedRunningTime="2026-10-30 02:59:13.071063732 +0000 UTC m=+1092.145234525"
Oct 30 02:59:13.174864 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.174827    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:59:13.175376 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:13.175337    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" containerID="cri-o://097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26" gracePeriod=30
Oct 30 02:59:14.040778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.040745    4214 generic.go:356] "Generic (PLEG): container finished" podID="7f072ce6-bff1-4b1c-bc0a-c23798419ebe" containerID="68cf81f186bc8b951d8f6d126e3a7c8b47a77da0475d22a2b44463272db4be51" exitCode=0
Oct 30 02:59:14.041202 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.040828    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px" event={"ID":"7f072ce6-bff1-4b1c-bc0a-c23798419ebe","Type":"ContainerDied","Data":"68cf81f186bc8b951d8f6d126e3a7c8b47a77da0475d22a2b44463272db4be51"}
Oct 30 02:59:14.274212 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.274176    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 02:59:14.275062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.275034    4214 cpu_manager.go:414] "containerMap: removing container" podUID="09d4f09b-29dd-45e3-be9a-60d4771bc838" containerName="revoke-keys"
Oct 30 02:59:14.275062 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.275054    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d4f09b-29dd-45e3-be9a-60d4771bc838" containerName="revoke-keys"
Oct 30 02:59:14.275275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.275245    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d4f09b-29dd-45e3-be9a-60d4771bc838" containerName="revoke-keys"
Oct 30 02:59:14.279478 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.279454    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.304637 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.304601    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 02:59:14.382158 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382117    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382192    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382393 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382356    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382405    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382529 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382513    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382606 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382592    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfkq\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382653 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382638    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382733    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.382784 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.382770    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484402 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484367    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484402 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484422    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484492    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484541    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484594    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484671 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484627    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484682    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484811 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484763    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfkq\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.484871 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.484807    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.485084 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.485057    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.485286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.485200    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.485286 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.485212    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.485434 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.485325    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.485434 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.485340    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.486880 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.486854    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.487014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.486990    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.505280 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.505255    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.506053 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.506034    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfkq\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:14.596461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:14.596375    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:15.047894 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.047867    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 02:59:15.049746 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:15.049721    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46021ffa_2f10_4e21_ac0c_cdf61d77b0af.slice/crio-ec4cb666a7b39059e0fb17306c195b13857785f4f8965b82b745e0601a3e2e7a WatchSource:0}: Error finding container ec4cb666a7b39059e0fb17306c195b13857785f4f8965b82b745e0601a3e2e7a: Status 404 returned error can't find the container with id ec4cb666a7b39059e0fb17306c195b13857785f4f8965b82b745e0601a3e2e7a
Oct 30 02:59:15.052648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.052607    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.052763 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.052689    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.052763 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.052732    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.189248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.188954    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 02:59:15.212304 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.211720    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.212968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.211604    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 02:59:15.220320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.220303    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 02:59:15.220443 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.220421    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.220655 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.220445    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 02:59:15.252014 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.251987    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:15.307069 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.306985    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjzz\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307281    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307324    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307477 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307455    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307529 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307493    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307631    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307722    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307899    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.307961 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.307941    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.409056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409026    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\" (UniqueName: \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\") pod \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") "
Oct 30 02:59:15.409243 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409188    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\") pod \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") "
Oct 30 02:59:15.409387 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409369    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\") pod \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\" (UID: \"7f072ce6-bff1-4b1c-bc0a-c23798419ebe\") "
Oct 30 02:59:15.409749 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409719    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.409858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409800    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjzz\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.409858 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409833    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.409959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409860    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.409959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409907    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.410135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.409971    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.410135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410000    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.410135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410048    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.410135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410102    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.410467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410169    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.410467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410258    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.410467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410288    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.410467 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410327    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.410733 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410711    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.410818 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410721    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca" pod "7f072ce6-bff1-4b1c-bc0a-c23798419ebe" (UID: "7f072ce6-bff1-4b1c-bc0a-c23798419ebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:15.410891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.410870    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.411030 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411011    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411152    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411314    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.411483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411465    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm88v\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.411689 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411632    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411091    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411414    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411166    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.411824 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.411712    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:15.412443 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.412422    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca" pod "7f072ce6-bff1-4b1c-bc0a-c23798419ebe" (UID: "7f072ce6-bff1-4b1c-bc0a-c23798419ebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:15.412545 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.412525    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b" pod "7f072ce6-bff1-4b1c-bc0a-c23798419ebe" (UID: "7f072ce6-bff1-4b1c-bc0a-c23798419ebe"). InnerVolumeSpecName "kube-api-access-s2h8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:15.413650 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.413632    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.414127 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.414109    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.429353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.429331    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.429501 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.429484    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjzz\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.512789 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.512762    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.512917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.512823    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.512917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.512843    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm88v\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.512917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.512895    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.512975    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513046    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513159    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513185    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513245 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513208    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.513491 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513307    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:15.513491 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513327    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2h8b\" (UniqueName: \"kubernetes.io/projected/7f072ce6-bff1-4b1c-bc0a-c23798419ebe-kube-api-access-s2h8b\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:15.513649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513619    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.514119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.513989    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.514119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.514011    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.514119 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.514073    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.514353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.514334    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.515516 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.515495    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.515752 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.515734    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.532589 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.532562    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.532746 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.532728    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm88v\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.542663 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.542640    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:15.550426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.550404    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:15.885766 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:15.885735    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d3d60e_10a7_4106_b959_ebcbc55ce921.slice/crio-c3eceab602ffb3a54e556037c3d569a7cc357fe80b454fca2fffd0630d3f01bd WatchSource:0}: Error finding container c3eceab602ffb3a54e556037c3d569a7cc357fe80b454fca2fffd0630d3f01bd: Status 404 returned error can't find the container with id c3eceab602ffb3a54e556037c3d569a7cc357fe80b454fca2fffd0630d3f01bd
Oct 30 02:59:15.887723 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.887698    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 02:59:15.888897 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.888859    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.889012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.888934    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.889012 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.888962    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.907248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.906623    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 02:59:15.911223 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.911189    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.911335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.911271    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:15.911335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:15.911304    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:16.115875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.115338    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" event={"ID":"41cc5e10-e962-4b86-8902-b986982ead92","Type":"ContainerStarted","Data":"8f61cbedeae0d3ea5051626651f80d71d7cf23f78d6206b4c1e1988e79c60b35"}
Oct 30 02:59:16.115875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.115404    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" event={"ID":"41cc5e10-e962-4b86-8902-b986982ead92","Type":"ContainerStarted","Data":"3bdb15a74ffe05074cce2e774f0315d84d3773a6b6fe71b7857cb1c8214ac6f4"}
Oct 30 02:59:16.122920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.120188    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" event={"ID":"b7d3d60e-10a7-4106-b959-ebcbc55ce921","Type":"ContainerStarted","Data":"5db97cb1864fa4951c0958cdff2a3462a2ba54c35f3baf25644e4ce42045fc6e"}
Oct 30 02:59:16.122920 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.120254    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" event={"ID":"b7d3d60e-10a7-4106-b959-ebcbc55ce921","Type":"ContainerStarted","Data":"c3eceab602ffb3a54e556037c3d569a7cc357fe80b454fca2fffd0630d3f01bd"}
Oct 30 02:59:16.124046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.124020    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" event={"ID":"46021ffa-2f10-4e21-ac0c-cdf61d77b0af","Type":"ContainerStarted","Data":"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"}
Oct 30 02:59:16.124222 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.124205    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" event={"ID":"46021ffa-2f10-4e21-ac0c-cdf61d77b0af","Type":"ContainerStarted","Data":"ec4cb666a7b39059e0fb17306c195b13857785f4f8965b82b745e0601a3e2e7a"}
Oct 30 02:59:16.126193 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.126169    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px" event={"ID":"7f072ce6-bff1-4b1c-bc0a-c23798419ebe","Type":"ContainerDied","Data":"fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c"}
Oct 30 02:59:16.126312 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.126217    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe23c580b4b0d2804c1a5e233311cfab3316bd5779e19b0a64a6fca7b229d43c"
Oct 30 02:59:16.126354 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.126299    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"
Oct 30 02:59:16.194250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.193680    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podStartSLOduration=1.193659614 podStartE2EDuration="1.193659614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.171674506 +0000 UTC m=+1095.245845301" watchObservedRunningTime="2026-10-30 02:59:16.193659614 +0000 UTC m=+1095.267830408"
Oct 30 02:59:16.297656 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.297588    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podStartSLOduration=2.29756846 podStartE2EDuration="2.29756846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.219943451 +0000 UTC m=+1095.294114243" watchObservedRunningTime="2026-10-30 02:59:16.29756846 +0000 UTC m=+1095.371739253"
Oct 30 02:59:16.304838 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.304757    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podStartSLOduration=1.304743958 podStartE2EDuration="1.304743958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.290171202 +0000 UTC m=+1095.364341995" watchObservedRunningTime="2026-10-30 02:59:16.304743958 +0000 UTC m=+1095.378914788"
Oct 30 02:59:16.548251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.546594    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:16.560253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.551401    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:16.600260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.597377    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:16.600260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.599687    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:16.600260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:16.599767    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:17.553054 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.553018    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.553648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.553615    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:17.554031 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.553503    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.554185 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.554165    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:17.609288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.609251    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.609542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:17.609511    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:18.312897 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.312851    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.437254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.398129    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.438217 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.430804    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 02:59:18.547091 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.547059    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.547384 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.547353    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:18.556568 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.556538    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.564828 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.564793    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:18.596770 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.596737    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.597022 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.596997    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:18.721622 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:18.719059    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c5622_fdb2_4ba2_b18c_f62ae4dd42d6.slice/crio-097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c5622_fdb2_4ba2_b18c_f62ae4dd42d6.slice/crio-conmon-097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 02:59:18.722100 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:18.722068    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c5622_fdb2_4ba2_b18c_f62ae4dd42d6.slice/crio-097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c5622_fdb2_4ba2_b18c_f62ae4dd42d6.slice/crio-conmon-097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 02:59:18.750451 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.749510    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.814362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:18.802343    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.321463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.321410    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.330255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.322445    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:59:19.610167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.607181    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.610167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.607292    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:19.610167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.608364    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.610167 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.609021    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:59:19.701264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.617201    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.701264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.617277    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:19.701264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.638624    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.701264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.638684    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:19.819269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.763409    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.819269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.763920    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:59:19.820969 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.820930    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.843355 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:19.840057    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:59:20.165255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.141658    4214 generic.go:356] "Generic (PLEG): container finished" podID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerID="097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26" exitCode=0
Oct 30 02:59:20.165841 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.163715    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" event={"ID":"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6","Type":"ContainerDied","Data":"097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26"}
Oct 30 02:59:20.331275 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.328540    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.593596 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.591496    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.593596 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.591597    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:20.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.607367    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.607478    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:20.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.609048    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.707336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.703547    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.707336 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.703620    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:20.899255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.873963    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.951456 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:20.951320    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.372409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.372355    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.560707 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.560665    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.560960 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.560930    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:21.613445 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.613397    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.629303 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.623116    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:59:21.681257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.631597    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.681257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.632991    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:21.681257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.633298    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.681257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.633338    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:21.715057 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.715007    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.744402 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.744363    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.744669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.744648    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.744802 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.744786    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.744917 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.744902    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.745065 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.745050    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.745199 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.745185    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.745422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.745406    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.752257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.745665    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.752533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.752510    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\") pod \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\" (UID: \"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6\") "
Oct 30 02:59:21.791502 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.791464    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:21.792049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.792023    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:21.792422 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.792401    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:21.810498 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.810465    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:21.811744 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.811716    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:21.877408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.877299    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:21.877408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.877334    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:21.877408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.877350    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:21.877408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.877364    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:21.877408 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.877379    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:21.902717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.902306    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:21.902717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.902373    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.916254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.906635    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "kube-api-access-2f25h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:21.916254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.909372    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 02:59:21.916254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.909544    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:21.942322 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.930662    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.942322 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.932360    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy" pod "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" (UID: "ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:21.972254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:21.963116    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.284320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.190685    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:22.284320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.281958    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f25h\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-kube-api-access-2f25h\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:22.284320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.281978    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:22.284320 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.281994    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:22.411262 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.409794    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.579256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.553892    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.591344 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.591302    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:22.591550 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.581609    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.591715 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.591691    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:22.619046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.615278    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.619046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.615332    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:22.619046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.615423    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.750309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.747255    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5" event={"ID":"ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6","Type":"ContainerDied","Data":"60c166b7461d23e3fcd0f3da94b1460bb433032ab66a9009ae6841fbc7ba7635"}
Oct 30 02:59:22.750309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.747345    4214 scope.go:122] "RemoveContainer" containerID="097ccc41b4b60df204247f809c1cdd23a9932b97ba46aa80b92b3f8ec55c0a26"
Oct 30 02:59:22.750309 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.747525    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"
Oct 30 02:59:22.818416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.818104    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8803-cfc2ad"]
Oct 30 02:59:22.820022 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.819980    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy"
Oct 30 02:59:22.825256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.820194    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy"
Oct 30 02:59:22.826875 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.825723    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7f072ce6-bff1-4b1c-bc0a-c23798419ebe" containerName="revoke-keys"
Oct 30 02:59:22.831350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.831277    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f072ce6-bff1-4b1c-bc0a-c23798419ebe" containerName="revoke-keys"
Oct 30 02:59:22.831873 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.831844    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" containerName="istio-proxy"
Oct 30 02:59:22.832017 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:22.832004    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f072ce6-bff1-4b1c-bc0a-c23798419ebe" containerName="revoke-keys"
Oct 30 02:59:23.052279 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.044392    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.052279 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.047692    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.189350 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.175998    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8803-cfc2ad"]
Oct 30 02:59:23.271253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.245989    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\") pod \"test-curl-8803-cfc2ad\" (UID: \"8e73de73-7615-47e5-bca2-ee3c60d52e4e\") " pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:23.271736 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.252866    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:23.273135 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.259327    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.273321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.273294    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.300440 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.295591    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 02:59:23.331850 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.331819    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.332056 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.332029    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.369349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.367450    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.369349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.367515    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.369349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.367621    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.369349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.367656    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.389128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.389089    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\") pod \"test-curl-8803-cfc2ad\" (UID: \"8e73de73-7615-47e5-bca2-ee3c60d52e4e\") " pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:23.489643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.410206    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:59:23.489643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.410284    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.489643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.440358    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.489643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.487997    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.489643 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.444250    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\") pod \"test-curl-8803-cfc2ad\" (UID: \"8e73de73-7615-47e5-bca2-ee3c60d52e4e\") " pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:23.491520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.490808    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:23.491520 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.491194    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-q5gc5"]
Oct 30 02:59:23.545181 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.545146    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.545521 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.545476    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:23.553699 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.553671    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.553981 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.553953    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:23.613267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.613181    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.613587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.613494    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:23.616066 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.615748    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.652484 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.652446    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.652780 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.652748    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.755538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.755344    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.755538 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:23.755404    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.053359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.052653    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.053359 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.053062    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.190649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.174091    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:24.190649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.174164    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.190649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.174683    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:24.190649 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.174720    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.500079 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.499981    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.594976 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.541545    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:24.599346 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.599296    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.614285 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.544130    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.614571 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.614536    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:24.614778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.585255    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.614932 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.614911    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:24.637397 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.586924    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:24.637624 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.637595    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.637783 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.617522    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.638204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.601617    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.638398 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:24.638377    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:25.288627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.288516    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.308308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.306920    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:25.308308 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.307754    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.637542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.557417    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.637542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.557481    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:25.637542 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.603083    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.675247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.675186    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:25.805847 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.800887    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.843255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.808487    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:25.876658 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.876622    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.906788 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.906749    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:25.907306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:25.890522    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:59:26.061263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.059481    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6" path="/var/lib/kubelet/pods/ce0c5622-fdb2-4ba2-b18c-f62ae4dd42d6/volumes"
Oct 30 02:59:26.061263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.060195    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8803-cfc2ad"]
Oct 30 02:59:26.339940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.339896    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:26.370331 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.341392    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:26.548622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.543089    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.548622 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.543158    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:26.568248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.561876    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.568248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.561936    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:26.613506 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.613407    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:26.613798 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.613752    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:26.615583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.615553    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:26.649079 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.649036    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.649369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.649341    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:26.700255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:26.688983    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:27.381395 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.381350    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:27.416668 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.381912    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:27.432378 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.432307    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 02:59:27.432940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.432909    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8803-cfc2ad" event={"ID":"8e73de73-7615-47e5-bca2-ee3c60d52e4e","Type":"ContainerStarted","Data":"33d3bcb26ba36f2abec18905014e2712aa79fbe9fef1fae75955d45c6fd1fe5a"}
Oct 30 02:59:27.545253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.544595    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.545253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.544661    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:27.563256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.552574    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.563256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.552631    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:27.598503 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.598464    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.598773 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.598742    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:27.619482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.619440    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:27.701774 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:27.701679    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:28.427510 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.427463    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:28.453002 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.452959    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:28.453441 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.453409    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.637893    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.669962    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.670016    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.709406    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.709473    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.709545    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.729296    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:28.743436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.729403    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:28.814259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.801503    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 02:59:28.833632 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.833599    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:28.909583 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:28.909465    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e2b0506d-openshift-default-dockercfg-m6c6b\""
Oct 30 02:59:29.287268 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.284175    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454465    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65p4q\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454552    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454632    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454685    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454733    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454764    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.454841    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.455080    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.455129    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.505256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.455139    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:29.603374 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.557437    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:29.660994 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.634764    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.665463 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.665429    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.712507 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.712466    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.712864 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.712713    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.713950 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.713904    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.714106 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.714089    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.714338 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.714314    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.714414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.714379    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.714628 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.714598    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65p4q\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.715146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.640789    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:29.715247 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.715184    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:29.733490 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.733455    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.733687 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.661597    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:29.734926 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.734900    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.735656    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.736201    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.736270    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.711734    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.738303    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.753720 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.709141    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.757845 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.658306    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"]
Oct 30 02:59:29.782273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.774022    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65p4q\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.790321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.666359    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:29.790321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.782747    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:29.790321 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.782973    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.834369 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.804334    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.887266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.887218    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\") pod \"e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") " pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:29.894436 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.894405    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:29.894678 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.894658    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"]
Oct 30 02:59:29.914273 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.895330    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:29.916461 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.916422    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:59:29.970252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.967130    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:30.001268 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:29.990181    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:59:30.080263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.077432    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qst\" (UniqueName: \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.080263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.077509    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.080263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.077603    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.205483 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.205392    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qst\" (UniqueName: \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.205721 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.205705    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.210271 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.206348    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.234608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.234572    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.495454 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.495338    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:30.498362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.498333    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.608049 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.608010    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:30.608655 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.608621    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:30.609197 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.609180    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:30.626876 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.626839    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:30.629508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.629483    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:30.680977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.645422    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:30.703891 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.703853    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:30.704353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.645623    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:30.704557 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.704532    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:30.735525 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.735486    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:30.740332 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.740295    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:30.750043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.748892    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qst\" (UniqueName: \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\") pod \"maas-api-revoke-sub-3cf58e08-glghw\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:30.908608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:30.908563    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:31.531346 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.531139    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:31.581368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.558974    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:31.581368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.559037    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:31.581368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.559390    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:31.581368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.559424    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:31.636647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.623640    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 02:59:31.636647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.625782    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:31.636647 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.625834    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:31.659832 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.659792    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:31.684132 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.683948    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:31.709913 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.691961    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:31.792426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.776458    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:31.792426 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:31.785157    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.109945    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wlzq\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110039    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110125    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110257    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110292    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110369    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110415    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110461    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.119216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.110547    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.277218 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.248970    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-93fe9de7-openshift-default-dockercfg-wv72m\""
Oct 30 02:59:32.389713 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.387130    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:32.400108 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.400069    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:32.402474 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.402444    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:32.536414 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.437770    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.538296 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.538211    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.538598 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.538562    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.538790 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.538758    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.538990 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.538953    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.539257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.539213    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.539533 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.539513    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlzq\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.539741 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.539701    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.539959 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.539940    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.540497 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.465490    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:59:32.559250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.552577    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:32.559250 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.552632    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:32.592259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.591398    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:32.634335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.631904    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:32.634335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.631961    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:32.634335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.632308    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:32.634335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.632338    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:32.748276 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.742006    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.765253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.749081    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.869587 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.750065    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.878368 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.791150    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:32.904027 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.815765    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.904249 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.816217    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:32.904363 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.829881    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:32.983334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:32.981883    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:33.001248 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.000613    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:33.091546 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.062860    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 02:59:33.261897 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.257592    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:33.403953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.392757    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlzq\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\") pod \"e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") " pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:33.417956 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.417896    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:59:33.427424 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.427358    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 02:59:33.427424 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.427414    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerStarted","Data":"ea795342bd6ca2441bb32b907394078d2a3f15b90f206d58a58177012e3bb011"}
Oct 30 02:59:33.427664 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.427445    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw" event={"ID":"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e","Type":"ContainerStarted","Data":"c17b40f4fcbbfaad5e2f918d709981b88d65f1551efe6345a7c0f8356eafc604"}
Oct 30 02:59:33.436036 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.436002    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:33.573207 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.573151    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:59:33.589029 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.588993    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:33.589335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.589294    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:33.609636 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.609604    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:33.609968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.609941    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:33.610878 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.610548    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:33.611063 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.611039    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:33.740710 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.623720    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:33.740710 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:33.625016    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:34.031799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.031766    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"]
Oct 30 02:59:34.035198 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.034779    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:59:34.375410 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:34.375361    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e4d8f4_da3e_4526_951a_bcc0aa4cd72e.slice/crio-c17b40f4fcbbfaad5e2f918d709981b88d65f1551efe6345a7c0f8356eafc604\": RecentStats: unable to find data in memory cache]"
Oct 30 02:59:34.528681 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.528640    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerStarted","Data":"00a1b2bf9f19e2e9b6fa58d333342c45a0f0f3adc14a8d5916c2d43b27db078d"}
Oct 30 02:59:34.547364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.546495    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:34.547364 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.546553    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:34.572867 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.572823    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:34.573046 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.572890    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:34.614551 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.612918    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podStartSLOduration=6.612897449 podStartE2EDuration="6.612897449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:34.573653115 +0000 UTC m=+1113.647823909" watchObservedRunningTime="2026-10-30 02:59:34.612897449 +0000 UTC m=+1113.687068243"
Oct 30 02:59:34.628518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.619891    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 02:59:34.628518 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.619944    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 02:59:34.983711 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:34.977304    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:35.104757 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.104710    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 02:59:35.109407 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.108668    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:35.109407 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.108752    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:35.109407 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.108803    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.543632    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.543689    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.547136    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" event={"ID":"6d2f77e9-1653-4d59-9eea-73d261d15e9a","Type":"ContainerStarted","Data":"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"}
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.547191    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" event={"ID":"6d2f77e9-1653-4d59-9eea-73d261d15e9a","Type":"ContainerStarted","Data":"14c783eba4fccce26c2099d080c362251501ccbdb46eb256aba8fa8f866d537d"}
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.550743    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 02:59:35.553305 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.550788    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 02:59:35.560536 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.559213    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw" event={"ID":"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e","Type":"ContainerStarted","Data":"162f143a183059202dff2061e2c1b0801a169348a4dd410089e16eb915c2c3ac"}
Oct 30 02:59:35.611423 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.610730    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" podStartSLOduration=4.6107095860000005 podStartE2EDuration="4.610709586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:35.606326397 +0000 UTC m=+1114.680497192" watchObservedRunningTime="2026-10-30 02:59:35.610709586 +0000 UTC m=+1114.684880382"
Oct 30 02:59:35.625391 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.625312    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:35.663335 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:35.663200    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw" podStartSLOduration=7.663180688 podStartE2EDuration="7.663180688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:35.650699827 +0000 UTC m=+1114.724870620" watchObservedRunningTime="2026-10-30 02:59:35.663180688 +0000 UTC m=+1114.737351482"
Oct 30 02:59:36.003254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.002587    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:36.578253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.577787    4214 generic.go:356] "Generic (PLEG): container finished" podID="13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" containerID="162f143a183059202dff2061e2c1b0801a169348a4dd410089e16eb915c2c3ac" exitCode=0
Oct 30 02:59:36.584251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.579793    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw" event={"ID":"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e","Type":"ContainerDied","Data":"162f143a183059202dff2061e2c1b0801a169348a4dd410089e16eb915c2c3ac"}
Oct 30 02:59:36.584251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.580807    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:36.584251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.582128    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 02:59:36.602251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.599925    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:36.602251 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.599979    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:36.651255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:36.650555    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:37.561301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.554325    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:37.561301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.554392    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:37.561301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.560332    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:37.561301 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.560394    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:37.666767 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.663910    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:37.666767 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.664002    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:37.711567 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.711536    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:37.746628 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:37.746598    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 02:59:38.196927 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.196898    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:38.206627 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.206604    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 02:59:38.355706 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.355058    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:38.529396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.528528    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\" (UniqueName: \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\") pod \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") "
Oct 30 02:59:38.529396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.528597    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\") pod \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") "
Oct 30 02:59:38.529396 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.528644    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\") pod \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\" (UID: \"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e\") "
Oct 30 02:59:38.530085 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.529793    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca" pod "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" (UID: "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:38.542392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.539867    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst" pod "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" (UID: "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e"). InnerVolumeSpecName "kube-api-access-w4qst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:38.542392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.541603    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca" pod "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" (UID: "13e4d8f4-da3e-4526-951a-bcc0aa4cd72e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:38.597716 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.596659    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:38.597716 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.596741    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:38.620849 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.620357    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:38.620849 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.620452    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:38.634383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.634312    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4qst\" (UniqueName: \"kubernetes.io/projected/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-kube-api-access-w4qst\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:38.634383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.634343    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:38.634383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.634359    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:38.722266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.718953    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"
Oct 30 02:59:38.722266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.719051    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw" event={"ID":"13e4d8f4-da3e-4526-951a-bcc0aa4cd72e","Type":"ContainerDied","Data":"c17b40f4fcbbfaad5e2f918d709981b88d65f1551efe6345a7c0f8356eafc604"}
Oct 30 02:59:38.722266 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.719106    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17b40f4fcbbfaad5e2f918d709981b88d65f1551efe6345a7c0f8356eafc604"
Oct 30 02:59:38.880272 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.879559    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:38.882343 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:38.882277    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 02:59:39.521030 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:39.520983    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 02:59:39.622648 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:39.622600    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:39.625418 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:39.624215    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:39.662460 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:39.662426    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:39.664177 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:39.664156    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 02:59:40.749651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:40.749605    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8803-cfc2ad" event={"ID":"8e73de73-7615-47e5-bca2-ee3c60d52e4e","Type":"ContainerStarted","Data":"c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b"}
Oct 30 02:59:40.789342 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:40.789290    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8803-cfc2ad" podStartSLOduration=4.403295808 podStartE2EDuration="18.789275954s" totalImagesPullingTime="14.385980146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:40.788563895 +0000 UTC m=+1119.862734690" watchObservedRunningTime="2026-10-30 02:59:40.789275954 +0000 UTC m=+1119.863446746"
Oct 30 02:59:41.874417 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.874387    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8803-cfc2ad"]
Oct 30 02:59:41.923011 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.919635    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8803-cfc2ad"]
Oct 30 02:59:41.989043 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.989009    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8803-386aeb"]
Oct 30 02:59:41.990032 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990010    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" containerName="test-curl-8803-cfc2ad"
Oct 30 02:59:41.990032 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990031    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" containerName="test-curl-8803-cfc2ad"
Oct 30 02:59:41.990172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990079    4214 cpu_manager.go:414] "containerMap: removing container" podUID="13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" containerName="revoke-keys"
Oct 30 02:59:41.990172 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990086    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" containerName="revoke-keys"
Oct 30 02:59:41.990252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990241    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" containerName="revoke-keys"
Oct 30 02:59:41.990288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.990260    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" containerName="test-curl-8803-cfc2ad"
Oct 30 02:59:41.994449 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:41.994431    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:42.009814 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.009788    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8803-386aeb"]
Oct 30 02:59:42.033315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.033277    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgqx\" (UniqueName: \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\") pod \"test-curl-8803-386aeb\" (UID: \"a1d6be96-e904-47cc-b3e6-f880c7122cca\") " pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:42.134792 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.134700    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgqx\" (UniqueName: \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\") pod \"test-curl-8803-386aeb\" (UID: \"a1d6be96-e904-47cc-b3e6-f880c7122cca\") " pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:42.167495 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.167463    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgqx\" (UniqueName: \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\") pod \"test-curl-8803-386aeb\" (UID: \"a1d6be96-e904-47cc-b3e6-f880c7122cca\") " pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:42.309869 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.309838    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:42.628716 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.628681    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8803-386aeb"]
Oct 30 02:59:42.646899 ip-10-0-136-171 kubenswrapper[4214]: W1030 02:59:42.646844    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d6be96_e904_47cc_b3e6_f880c7122cca.slice/crio-1f9e00b0f2c23c1e1bad263435515e0f757db911cebfa09e7188e5f8d391a957 WatchSource:0}: Error finding container 1f9e00b0f2c23c1e1bad263435515e0f757db911cebfa09e7188e5f8d391a957: Status 404 returned error can't find the container with id 1f9e00b0f2c23c1e1bad263435515e0f757db911cebfa09e7188e5f8d391a957
Oct 30 02:59:42.771704 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.771672    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8803-386aeb" event={"ID":"a1d6be96-e904-47cc-b3e6-f880c7122cca","Type":"ContainerStarted","Data":"7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60"}
Oct 30 02:59:42.771872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.771722    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8803-386aeb" event={"ID":"a1d6be96-e904-47cc-b3e6-f880c7122cca","Type":"ContainerStarted","Data":"1f9e00b0f2c23c1e1bad263435515e0f757db911cebfa09e7188e5f8d391a957"}
Oct 30 02:59:42.771872 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.771805    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8803-cfc2ad" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" containerName="test-curl-8803-cfc2ad" containerID="cri-o://c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b" gracePeriod=2
Oct 30 02:59:42.810298 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:42.810246    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8803-386aeb" podStartSLOduration=1.810210469 podStartE2EDuration="1.810210469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:42.809798366 +0000 UTC m=+1121.883969158" watchObservedRunningTime="2026-10-30 02:59:42.810210469 +0000 UTC m=+1121.884381263"
Oct 30 02:59:42.823996 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:42.823964    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-cfc2ad\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:42.875552 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:42.844493    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-cfc2ad\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:43.401264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:43.397137    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:59:43.401264 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:43.397420    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy" containerID="cri-o://09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349" gracePeriod=30
Oct 30 02:59:43.917840 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:43.916847    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8803-386aeb"]
Oct 30 02:59:43.925646 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:43.925613    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8803-386aeb"]
Oct 30 02:59:44.486669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.486636    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 02:59:44.487512 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.487479    4214 cpu_manager.go:414] "containerMap: removing container" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" containerName="test-curl-8803-386aeb"
Oct 30 02:59:44.487512 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.487498    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" containerName="test-curl-8803-386aeb"
Oct 30 02:59:44.487691 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.487645    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" containerName="test-curl-8803-386aeb"
Oct 30 02:59:44.492267 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.492219    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.525739 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.525704    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.525877 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.525763    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.525877 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.525868    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.525900    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth7v\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526023 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.525950    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.526039    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526144 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.526091    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.526158    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.526288 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.526183    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.529492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.529468    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 02:59:44.627778 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627746    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.627968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627785    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth7v\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.627968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627822    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.627968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627882    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.627968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627917    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.627968 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627963    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.627987    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628059    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628191 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628104    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628475 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628475    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628532 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628512    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628695 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628603    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.628899 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.628876    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.630362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.630341    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.630572 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.630556    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.662260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.659636    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.662260 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.660072    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth7v\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.837265 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.828821    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:44.906632 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:44.906458    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8803-386aeb" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" containerName="test-curl-8803-386aeb" containerID="cri-o://7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60" gracePeriod=2
Oct 30 02:59:44.950628 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:44.947872    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-386aeb\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:45.291653 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.291626    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:45.357352 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.357263    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\") pod \"8e73de73-7615-47e5-bca2-ee3c60d52e4e\" (UID: \"8e73de73-7615-47e5-bca2-ee3c60d52e4e\") "
Oct 30 02:59:45.360039 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.360011    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f" pod "8e73de73-7615-47e5-bca2-ee3c60d52e4e" (UID: "8e73de73-7615-47e5-bca2-ee3c60d52e4e"). InnerVolumeSpecName "kube-api-access-ldl6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:45.473329 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.460315    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl6f\" (UniqueName: \"kubernetes.io/projected/8e73de73-7615-47e5-bca2-ee3c60d52e4e-kube-api-access-ldl6f\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:45.561833 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.561806    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 02:59:45.568099 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.568065    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.568206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.568134    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.568206 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.568166    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:59:45.917128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.917042    4214 generic.go:356] "Generic (PLEG): container finished" podID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" containerID="c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b" exitCode=137
Oct 30 02:59:45.917128 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.917093    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:45.917373 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.917133    4214 scope.go:122] "RemoveContainer" containerID="c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b"
Oct 30 02:59:45.919166 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.919134    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" event={"ID":"d74ec34e-e120-4de0-b38b-3f396ed51414","Type":"ContainerStarted","Data":"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"}
Oct 30 02:59:45.919315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.919191    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" event={"ID":"d74ec34e-e120-4de0-b38b-3f396ed51414","Type":"ContainerStarted","Data":"8ec4a33da3bd6e3ca8274d13aa4499cbbbda89a50c0d359f92660d6f4a57f2db"}
Oct 30 02:59:45.931549 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.931530    4214 scope.go:122] "RemoveContainer" containerID="c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b"
Oct 30 02:59:45.932356 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:45.932327    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b\": container with ID starting with c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b not found: ID does not exist" containerID="c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b"
Oct 30 02:59:45.932439 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.932366    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b"} err="failed to get container status \"c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b\": rpc error: code = NotFound desc = could not find container \"c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b\": container with ID starting with c1747f38e70ecbd50bed7646f893d985f345cfe38e71d14877beee165600ba9b not found: ID does not exist"
Oct 30 02:59:45.992381 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:45.992336    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-cfc2ad\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" pod="opendatahub/test-curl-8803-cfc2ad"
Oct 30 02:59:45.992772 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:45.992673    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podStartSLOduration=1.992659169 podStartE2EDuration="1.992659169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:45.977144174 +0000 UTC m=+1125.051314979" watchObservedRunningTime="2026-10-30 02:59:45.992659169 +0000 UTC m=+1125.066829961"
Oct 30 02:59:46.831416 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:46.831378    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:46.833028 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:46.832996    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:46.833146 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:46.833048    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:47.119835 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.119795    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e73de73-7615-47e5-bca2-ee3c60d52e4e" path="/var/lib/kubelet/pods/8e73de73-7615-47e5-bca2-ee3c60d52e4e/volumes"
Oct 30 02:59:47.384799 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.384725    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:47.515380 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.515351    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\" (UniqueName: \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\") pod \"a1d6be96-e904-47cc-b3e6-f880c7122cca\" (UID: \"a1d6be96-e904-47cc-b3e6-f880c7122cca\") "
Oct 30 02:59:47.517482 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.517457    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx" pod "a1d6be96-e904-47cc-b3e6-f880c7122cca" (UID: "a1d6be96-e904-47cc-b3e6-f880c7122cca"). InnerVolumeSpecName "kube-api-access-dfgqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:47.617152 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.617118    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfgqx\" (UniqueName: \"kubernetes.io/projected/a1d6be96-e904-47cc-b3e6-f880c7122cca-kube-api-access-dfgqx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:47.830965 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.830932    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:47.831134 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.830989    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:47.988488 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.988454    4214 generic.go:356] "Generic (PLEG): container finished" podID="a1d6be96-e904-47cc-b3e6-f880c7122cca" containerID="7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60" exitCode=137
Oct 30 02:59:47.988652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.988510    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:47.988652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.988604    4214 scope.go:122] "RemoveContainer" containerID="7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60"
Oct 30 02:59:47.999912 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:47.999891    4214 scope.go:122] "RemoveContainer" containerID="7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60"
Oct 30 02:59:48.000207 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:48.000186    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60\": container with ID starting with 7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60 not found: ID does not exist" containerID="7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60"
Oct 30 02:59:48.000287 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.000221    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60"} err="failed to get container status \"7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60\": rpc error: code = NotFound desc = could not find container \"7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60\": container with ID starting with 7c835e37e63ac943ab07b5aeb5987cd77f654a7d367d627ef0859af1399bca60 not found: ID does not exist"
Oct 30 02:59:48.011852 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:48.011820    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-386aeb\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:48.725450 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.725386    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:59:48.824249 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:48.819474    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-386aeb\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:48.874588 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.874550    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.874821 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.874802    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.875435 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.875411    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.881291 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.875624    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.909738 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.909704    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.920365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.920329    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.920674 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.920646    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.923341 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.923309    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:48.939258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.923546    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\") pod \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\" (UID: \"dd1c088d-4fdf-4f6c-92f4-40f5809564a6\") "
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.878466    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.963566    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.925030    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.925207    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.935163    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.940429    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:48.964035    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.011343    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:49.029353 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.011475    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "kube-api-access-xxdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:49.046392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.033637    4214 generic.go:356] "Generic (PLEG): container finished" podID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerID="09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349" exitCode=0
Oct 30 02:59:49.046392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.033766    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" event={"ID":"dd1c088d-4fdf-4f6c-92f4-40f5809564a6","Type":"ContainerDied","Data":"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"}
Oct 30 02:59:49.046392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.033817    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v" event={"ID":"dd1c088d-4fdf-4f6c-92f4-40f5809564a6","Type":"ContainerDied","Data":"fa58e043925eac9b1b9b2c4aaae3b637dc8642e36ac7ef6cd259fd3ee9a3bc9f"}
Oct 30 02:59:49.046392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.033848    4214 scope.go:122] "RemoveContainer" containerID="09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"
Oct 30 02:59:49.046392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.033972    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"
Oct 30 02:59:49.073514 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:49.073447    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8803-386aeb\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" pod="opendatahub/test-curl-8803-386aeb"
Oct 30 02:59:49.074064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.074021    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:59:49.075712 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075685    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075717    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075732    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075745    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdxf\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-kube-api-access-xxdxf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075760    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075774    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075786    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.075820 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.075798    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.081269 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.080022    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token" pod "dd1c088d-4fdf-4f6c-92f4-40f5809564a6" (UID: "dd1c088d-4fdf-4f6c-92f4-40f5809564a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:49.086940 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.086924    4214 scope.go:122] "RemoveContainer" containerID="09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"
Oct 30 02:59:49.087397 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:49.087379    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349\": container with ID starting with 09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349 not found: ID does not exist" containerID="09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"
Oct 30 02:59:49.087495 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.087478    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349"} err="failed to get container status \"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349\": rpc error: code = NotFound desc = could not find container \"09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349\": container with ID starting with 09b26682e061689f90820962063014df6b6122765c736c1c572f37bcd1752349 not found: ID does not exist"
Oct 30 02:59:49.088173 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.088152    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d6be96-e904-47cc-b3e6-f880c7122cca" path="/var/lib/kubelet/pods/a1d6be96-e904-47cc-b3e6-f880c7122cca/volumes"
Oct 30 02:59:49.177601 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.177568    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1c088d-4fdf-4f6c-92f4-40f5809564a6-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:49.410977 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.410939    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:59:49.423156 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.423118    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-v5g8v"]
Oct 30 02:59:49.823651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.823617    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"]
Oct 30 02:59:49.824492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.824474    4214 cpu_manager.go:414] "containerMap: removing container" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy"
Oct 30 02:59:49.824492 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.824493    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy"
Oct 30 02:59:49.824651 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.824617    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" containerName="istio-proxy"
Oct 30 02:59:49.828324 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.828306    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.831187 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.831166    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:49.831271 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.831217    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:49.859606 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.856528    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 02:59:49.859606 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.856597    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 02:59:49.885383 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.885258    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"]
Oct 30 02:59:49.886681 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.886653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.887064 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.887044    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.887216 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.887202    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdht2\" (UniqueName: \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.989283 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.989248    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.989469 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.989450    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.989526 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.989511    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdht2\" (UniqueName: \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.990123 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.990101    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:49.991580 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:49.991558    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:50.009051 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:50.009026    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdht2\" (UniqueName: \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\") pod \"maas-api-revoke-sub-d5886317-27gpv\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:50.179717 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:50.179631    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:50.706697 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:50.706657    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"]
Oct 30 02:59:50.830975 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:50.830944    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:50.831349 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:50.831011    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:51.069955 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:51.069920    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1c088d-4fdf-4f6c-92f4-40f5809564a6" path="/var/lib/kubelet/pods/dd1c088d-4fdf-4f6c-92f4-40f5809564a6/volumes"
Oct 30 02:59:51.081890 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:51.081856    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv" event={"ID":"a8fcdafb-a368-4f00-815e-ba3be4d39723","Type":"ContainerStarted","Data":"06640fecc815752cf614c89549dfb32522a84b7009e067d76d6ab81a5a272084"}
Oct 30 02:59:51.846252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:51.841686    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:51.846252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:51.841746    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:52.414252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.405770    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv" event={"ID":"a8fcdafb-a368-4f00-815e-ba3be4d39723","Type":"ContainerStarted","Data":"7d5acac63389ea6bbac4d7fd14e2cd1a8cfeddabc105f39de888fdc53613bc3c"}
Oct 30 02:59:52.513252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.512413    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:52.627652 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.627602    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.834302 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.834262    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:52.834577 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.834543    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:52.921389 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.921338    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:52.964859 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:52.964814    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.387896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.387423    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.387896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.387496    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.510310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.509605    4214 generic.go:356] "Generic (PLEG): container finished" podID="a8fcdafb-a368-4f00-815e-ba3be4d39723" containerID="7d5acac63389ea6bbac4d7fd14e2cd1a8cfeddabc105f39de888fdc53613bc3c" exitCode=0
Oct 30 02:59:53.510310 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.509821    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv" event={"ID":"a8fcdafb-a368-4f00-815e-ba3be4d39723","Type":"ContainerDied","Data":"7d5acac63389ea6bbac4d7fd14e2cd1a8cfeddabc105f39de888fdc53613bc3c"}
Oct 30 02:59:53.539718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.531600    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.539718 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.532114    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:59:53.839357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.654352    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.839357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.656411    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:59:53.839357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.802343    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.839357 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.802405    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.897908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.892127    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:53.897908 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.892195    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:53.949284 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.949222    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.950259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:53.950215    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:59:54.000556 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.000511    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:54.000845 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.000814    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.123801 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.123699    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:54.186258 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.185654    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.573252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.560466    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.659365 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:54.659316    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.021896 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.021785    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:55.022494 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.022440    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:55.548315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.283216    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.548315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.299004    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 02:59:55.548315 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.362288    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.618281 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.603513    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.681530 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.681475    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.977274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:55.949558    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:56.261887 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.254104    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:56.652254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.624106    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:56.741508 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.645575    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:56.761259 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.742214    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 02:59:56.761528 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.654757    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 02:59:56.819253 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.816876    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:57.001306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.977920    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:57.001306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.977987    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:57.001306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.980398    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:57.001306 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:56.980462    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:57.223543 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.223507    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:57.445252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.441030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\") pod \"a8fcdafb-a368-4f00-815e-ba3be4d39723\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") "
Oct 30 02:59:57.445252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.441253    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\") pod \"a8fcdafb-a368-4f00-815e-ba3be4d39723\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") "
Oct 30 02:59:57.445252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.441366    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\" (UniqueName: \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\") pod \"a8fcdafb-a368-4f00-815e-ba3be4d39723\" (UID: \"a8fcdafb-a368-4f00-815e-ba3be4d39723\") "
Oct 30 02:59:57.445252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.443156    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca" pod "a8fcdafb-a368-4f00-815e-ba3be4d39723" (UID: "a8fcdafb-a368-4f00-815e-ba3be4d39723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:59:57.534025 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.533987    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv" event={"ID":"a8fcdafb-a368-4f00-815e-ba3be4d39723","Type":"ContainerDied","Data":"06640fecc815752cf614c89549dfb32522a84b7009e067d76d6ab81a5a272084"}
Oct 30 02:59:57.534282 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.534265    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06640fecc815752cf614c89549dfb32522a84b7009e067d76d6ab81a5a272084"
Oct 30 02:59:57.534453 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.534440    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"
Oct 30 02:59:57.553257 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.544950    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:57.646392 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.646351    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 02:59:57.692252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.664674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2" pod "a8fcdafb-a368-4f00-815e-ba3be4d39723" (UID: "a8fcdafb-a368-4f00-815e-ba3be4d39723"). InnerVolumeSpecName "kube-api-access-wdht2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:59:57.692252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.683343    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 02:59:57.692252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.684470    4214 cpu_manager.go:414] "containerMap: removing container" podUID="a8fcdafb-a368-4f00-815e-ba3be4d39723" containerName="revoke-keys"
Oct 30 02:59:57.692252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.684494    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fcdafb-a368-4f00-815e-ba3be4d39723" containerName="revoke-keys"
Oct 30 02:59:57.692252 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.684699    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fcdafb-a368-4f00-815e-ba3be4d39723" containerName="revoke-keys"
Oct 30 02:59:57.730313 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.704604    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:57.739263 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.705351    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca" pod "a8fcdafb-a368-4f00-815e-ba3be4d39723" (UID: "a8fcdafb-a368-4f00-815e-ba3be4d39723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:59:57.800256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.797224    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:57.800256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.797669    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:57.800256 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.797996    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:57.816254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.812686    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e2b0506d-serving-cert\""
Oct 30 02:59:57.847669 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.825483    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:57.855489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.851570    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 02:59:57.855489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.851794    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 02:59:57.855489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.851924    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdht2\" (UniqueName: \"kubernetes.io/projected/a8fcdafb-a368-4f00-815e-ba3be4d39723-kube-api-access-wdht2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:57.855489 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.852125    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fcdafb-a368-4f00-815e-ba3be4d39723-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 02:59:57.929645 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:57.897371    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 02:59:58.004300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.003387    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.004300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.003585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.004300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.003707    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686bb\" (UniqueName: \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.004300 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.003777    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.116998    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.117170    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.117317    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.117416    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686bb\" (UniqueName: \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.118449    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:58.118557    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-serving-cert: secret "maas-api-e2e-ait-e2b0506d-serving-cert" not found
Oct 30 02:59:58.124274 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:58.118620    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls podName:75b933e0-e300-48f4-8e0c-64290e7c74c9 nodeName:}" failed. No retries permitted until 2026-10-30 02:59:58.618600913 +0000 UTC m=+1137.692771690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls") pod "maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" (UID: "75b933e0-e300-48f4-8e0c-64290e7c74c9") : secret "maas-api-e2e-ait-e2b0506d-serving-cert" not found
Oct 30 02:59:58.446597 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.446337    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 02:59:58.477970 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.477934    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.500400 ip-10-0-136-171 kubenswrapper[4214]: E1030 02:59:58.494341    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fcdafb_a368_4f00_815e_ba3be4d39723.slice/crio-06640fecc815752cf614c89549dfb32522a84b7009e067d76d6ab81a5a272084\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fcdafb_a368_4f00_815e_ba3be4d39723.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 02:59:58.670829 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.598084    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686bb\" (UniqueName: \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.743334 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.704346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.829409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.803323    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:58.829409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.803704    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 02:59:58.829409 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.803969    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:58.865805 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.864007    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 02:59:58.937254 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.911803    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") pod \"maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:58.992953 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:58.992846    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 02:59:59.241255 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.222573    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 02:59:59.874182 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.868560    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:59.879362 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.879224    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 02:59:59.898608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.885646    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:59.898608 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.898442    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 02:59:59.918168 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.918143    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 02:59:59.946204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.939735    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:59.946204 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.939782    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:59.999706 ip-10-0-136-171 kubenswrapper[4214]: I1030 02:59:59.999679    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:00:00.061807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.054074    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"]
Oct 30 03:00:00.063101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.063030    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1702e439-6v5bv"]
Oct 30 03:00:00.110293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.110019    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 03:00:00.112290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.112195    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 03:00:00.168365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.168334    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:00.176134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.176111    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.177090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.177070    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:00.184732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.183061    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.195695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.195664    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.195944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.195740    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.195944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.195913    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfsp\" (UniqueName: \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.196191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.196001    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62kf\" (UniqueName: \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.197253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.197177    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:00.203740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.202947    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.206086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.206051    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:00.227086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.226985    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.279292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.252494    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:00.330261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.304941    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfsp\" (UniqueName: \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.354022    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62kf\" (UniqueName: \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.354519    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.354587    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:00.349349    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b933e0_e300_48f4_8e0c_64290e7c74c9.slice/crio-b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592 WatchSource:0}: Error finding container b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592: Status 404 returned error can't find the container with id b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.351116    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:00.372256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.365064    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.432140    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.442731    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.458222    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.458718    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxq2l\" (UniqueName: \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.458928    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.464432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.458966    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfc9f\" (UniqueName: \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.517102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.516781    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.520161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.520075    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 03:00:00.554323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.554211    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfsp\" (UniqueName: \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\") pod \"maas-api-key-cleanup-29888820-l9t27\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.558840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.558811    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62kf\" (UniqueName: \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\") pod \"maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.560276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.560223    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.560482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.560464    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxq2l\" (UniqueName: \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.560752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.560734    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.560869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.560856    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfc9f\" (UniqueName: \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.562132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.562110    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.563132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.563110    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.569727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.569706    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:00.584454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.584171    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfc9f\" (UniqueName: \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.584454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.584420    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxq2l\" (UniqueName: \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.611316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.611293    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"]
Oct 30 03:00:00.617579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.617562    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.628545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.628519    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"]
Oct 30 03:00:00.634892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.634870    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:00.668283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.664393    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.682263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.681620    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdk6h\" (UniqueName: \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.682263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.681845    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.728921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.728819    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:00.769288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.769259    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:00.783832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.783126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.783832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.783290    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.783832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.783458    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdk6h\" (UniqueName: \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.784075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.784046    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.786475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.786451    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.805942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.805911    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdk6h\" (UniqueName: \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\") pod \"maas-api-revoke-sub-7347e427-5vm8f\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:00.806203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.806180    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:00.837837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.837567    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:00.940798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:00.940572    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:01.236753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.236129    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e162a449-4ed6-4e1b-a84c-a3df776bc96a" path="/var/lib/kubelet/pods/e162a449-4ed6-4e1b-a84c-a3df776bc96a/volumes"
Oct 30 03:00:01.236753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.236683    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" event={"ID":"75b933e0-e300-48f4-8e0c-64290e7c74c9","Type":"ContainerStarted","Data":"b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592"}
Oct 30 03:00:01.237815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.237774    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:01.376057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.375824    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:01.501717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.501686    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:01.513080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.513059    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:01.513679 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:01.513655    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e956e08_0d5d_485e_b4d6_70f9b6aa251c.slice/crio-45c68c2264f62aaf5297e93a61d5a6a9ebc3862c6c412259ed688b3d1114691f WatchSource:0}: Error finding container 45c68c2264f62aaf5297e93a61d5a6a9ebc3862c6c412259ed688b3d1114691f: Status 404 returned error can't find the container with id 45c68c2264f62aaf5297e93a61d5a6a9ebc3862c6c412259ed688b3d1114691f
Oct 30 03:00:01.657142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:01.657103    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"]
Oct 30 03:00:02.245239 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.245193    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerStarted","Data":"9b1404519e1875778aa1eea4376bcb4f482ec9a19a3eaa91e53c105786e526d5"}
Oct 30 03:00:02.245644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.245264    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerStarted","Data":"96f2fa35517fb6c4f08ac72d00e58c44b2ed722f1ff178ea48d1e41f68d7a549"}
Oct 30 03:00:02.247077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.247051    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerStarted","Data":"68e2dd364f56068ebe49ec2d32fb0788b5ef73d2754a88d49bbb321a93432980"}
Oct 30 03:00:02.247200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.247108    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerStarted","Data":"5265fd52670463821085fb6a8449c9679598cedb2e3ec645e43e379d33513636"}
Oct 30 03:00:02.249099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.249071    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" event={"ID":"75b933e0-e300-48f4-8e0c-64290e7c74c9","Type":"ContainerStarted","Data":"082ae025149388a267ddd5c179e93b327839bb8c7e8ea34a5aae57eb178bb86e"}
Oct 30 03:00:02.249212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.249150    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 03:00:02.250777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.250756    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerStarted","Data":"632d3b8f48192015974b88460e874ca27030c0953534d3ab0ef006112480b1be"}
Oct 30 03:00:02.250881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.250802    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerStarted","Data":"859142bf678b88bd010918f29875ab46735f06783e718054d03ca9afef34ce76"}
Oct 30 03:00:02.252633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.252608    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerStarted","Data":"3b5d79fd7fbe5dedbab7401e56555da5524c98f1a2c90fb602bc500b35bf18f6"}
Oct 30 03:00:02.252731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.252665    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerStarted","Data":"45c68c2264f62aaf5297e93a61d5a6a9ebc3862c6c412259ed688b3d1114691f"}
Oct 30 03:00:02.260618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.260582    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f" event={"ID":"994fd1b9-4e15-49a3-9446-5630966e7517","Type":"ContainerStarted","Data":"1b01618b36636e2e4686f8985022f165ae4eee93b5c19cb10c1263c929463a3d"}
Oct 30 03:00:02.260726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.260686    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f" event={"ID":"994fd1b9-4e15-49a3-9446-5630966e7517","Type":"ContainerStarted","Data":"697d32b84560f84b38a64ce5fcb187802536087ed6f6fda0bf0b8d9791646282"}
Oct 30 03:00:02.290981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.290933    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" podStartSLOduration=2.29092157 podStartE2EDuration="2.29092157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.285631142 +0000 UTC m=+1141.359801936" watchObservedRunningTime="2026-10-30 03:00:02.29092157 +0000 UTC m=+1141.365092362"
Oct 30 03:00:02.326238 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.326169    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" podStartSLOduration=4.788952761 podStartE2EDuration="5.326156486s" totalImagesPullingTime="537.203725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.32589752 +0000 UTC m=+1141.400068312" watchObservedRunningTime="2026-10-30 03:00:02.326156486 +0000 UTC m=+1141.400327278"
Oct 30 03:00:02.367480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.367430    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f" podStartSLOduration=2.367416995 podStartE2EDuration="2.367416995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.364854106 +0000 UTC m=+1141.439024900" watchObservedRunningTime="2026-10-30 03:00:02.367416995 +0000 UTC m=+1141.441587839"
Oct 30 03:00:02.407525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.407412    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" podStartSLOduration=2.40739781 podStartE2EDuration="2.40739781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.402729966 +0000 UTC m=+1141.476900760" watchObservedRunningTime="2026-10-30 03:00:02.40739781 +0000 UTC m=+1141.481568603"
Oct 30 03:00:02.477275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.468688    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" podStartSLOduration=2.468665828 podStartE2EDuration="2.468665828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.441666437 +0000 UTC m=+1141.515837230" watchObservedRunningTime="2026-10-30 03:00:02.468665828 +0000 UTC m=+1141.542836623"
Oct 30 03:00:02.530600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.526793    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" podStartSLOduration=2.526770327 podStartE2EDuration="2.526770327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:02.491651882 +0000 UTC m=+1141.565822689" watchObservedRunningTime="2026-10-30 03:00:02.526770327 +0000 UTC m=+1141.600941120"
Oct 30 03:00:02.629285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.629217    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:02.652563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.651096    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.682115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.678452    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:02.682115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.681574    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93fe9de7-serving-cert\""
Oct 30 03:00:02.725099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.725063    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.725267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.725212    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.725414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.725398    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.725474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.725449    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swp9x\" (UniqueName: \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.864303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.860548    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.864303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.860645    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swp9x\" (UniqueName: \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.864303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.860910    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.864303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.861021    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.864303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.862881    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.893148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.889965    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swp9x\" (UniqueName: \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.894812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.894523    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:02.904966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:02.904891    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\") pod \"maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:03.000710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.000567    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:03.273374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.273282    4214 generic.go:356] "Generic (PLEG): container finished" podID="1954967c-5058-4199-88db-d758095c5a73" containerID="632d3b8f48192015974b88460e874ca27030c0953534d3ab0ef006112480b1be" exitCode=60
Oct 30 03:00:03.273822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.273371    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerDied","Data":"632d3b8f48192015974b88460e874ca27030c0953534d3ab0ef006112480b1be"}
Oct 30 03:00:03.273822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.273681    4214 scope.go:122] "RemoveContainer" containerID="632d3b8f48192015974b88460e874ca27030c0953534d3ab0ef006112480b1be"
Oct 30 03:00:03.275994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.275953    4214 generic.go:356] "Generic (PLEG): container finished" podID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerID="3b5d79fd7fbe5dedbab7401e56555da5524c98f1a2c90fb602bc500b35bf18f6" exitCode=60
Oct 30 03:00:03.276132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.276114    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerDied","Data":"3b5d79fd7fbe5dedbab7401e56555da5524c98f1a2c90fb602bc500b35bf18f6"}
Oct 30 03:00:03.276452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.276438    4214 scope.go:122] "RemoveContainer" containerID="3b5d79fd7fbe5dedbab7401e56555da5524c98f1a2c90fb602bc500b35bf18f6"
Oct 30 03:00:03.279676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.279649    4214 generic.go:356] "Generic (PLEG): container finished" podID="994fd1b9-4e15-49a3-9446-5630966e7517" containerID="1b01618b36636e2e4686f8985022f165ae4eee93b5c19cb10c1263c929463a3d" exitCode=0
Oct 30 03:00:03.279764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.279681    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f" event={"ID":"994fd1b9-4e15-49a3-9446-5630966e7517","Type":"ContainerDied","Data":"1b01618b36636e2e4686f8985022f165ae4eee93b5c19cb10c1263c929463a3d"}
Oct 30 03:00:03.281833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.281811    4214 generic.go:356] "Generic (PLEG): container finished" podID="771039f7-3134-4c81-af26-38a52d5bb76f" containerID="9b1404519e1875778aa1eea4376bcb4f482ec9a19a3eaa91e53c105786e526d5" exitCode=60
Oct 30 03:00:03.281940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.281848    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerDied","Data":"9b1404519e1875778aa1eea4376bcb4f482ec9a19a3eaa91e53c105786e526d5"}
Oct 30 03:00:03.282162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.282138    4214 scope.go:122] "RemoveContainer" containerID="9b1404519e1875778aa1eea4376bcb4f482ec9a19a3eaa91e53c105786e526d5"
Oct 30 03:00:03.283765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.283739    4214 generic.go:356] "Generic (PLEG): container finished" podID="7a080b24-8407-45fe-8eff-86d30034ee70" containerID="68e2dd364f56068ebe49ec2d32fb0788b5ef73d2754a88d49bbb321a93432980" exitCode=60
Oct 30 03:00:03.283882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.283814    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerDied","Data":"68e2dd364f56068ebe49ec2d32fb0788b5ef73d2754a88d49bbb321a93432980"}
Oct 30 03:00:03.284219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.284202    4214 scope.go:122] "RemoveContainer" containerID="68e2dd364f56068ebe49ec2d32fb0788b5ef73d2754a88d49bbb321a93432980"
Oct 30 03:00:03.332631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:03.332610    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:04.297787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.297751    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerStarted","Data":"b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"}
Oct 30 03:00:04.303977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.303948    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" event={"ID":"23ef435d-2d75-4dbd-8159-f7b5537576e0","Type":"ContainerStarted","Data":"ce431cb00716ba4d32efdc37e00862feace98790383667fc4b207921c8af70aa"}
Oct 30 03:00:04.304185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.304168    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" event={"ID":"23ef435d-2d75-4dbd-8159-f7b5537576e0","Type":"ContainerStarted","Data":"10cb02b45c879a6f74a65f24051a644f3cf247767cd151bd51c16f2ecd32de09"}
Oct 30 03:00:04.305003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.304982    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:04.307723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.307690    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerStarted","Data":"fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"}
Oct 30 03:00:04.311974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.311950    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerStarted","Data":"63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"}
Oct 30 03:00:04.314205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.314180    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerStarted","Data":"85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"}
Oct 30 03:00:04.494924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.494870    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" podStartSLOduration=2.048477679 podStartE2EDuration="2.494851183s" totalImagesPullingTime="446.373504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:04.486858857 +0000 UTC m=+1143.561029649" watchObservedRunningTime="2026-10-30 03:00:04.494851183 +0000 UTC m=+1143.569021977"
Oct 30 03:00:04.766045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.766012    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:04.879559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.879459    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\") pod \"994fd1b9-4e15-49a3-9446-5630966e7517\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") "
Oct 30 03:00:04.879927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.879894    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\") pod \"994fd1b9-4e15-49a3-9446-5630966e7517\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") "
Oct 30 03:00:04.880054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.880037    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\" (UniqueName: \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\") pod \"994fd1b9-4e15-49a3-9446-5630966e7517\" (UID: \"994fd1b9-4e15-49a3-9446-5630966e7517\") "
Oct 30 03:00:04.880347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.880315    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca" pod "994fd1b9-4e15-49a3-9446-5630966e7517" (UID: "994fd1b9-4e15-49a3-9446-5630966e7517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:04.881023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.881000    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:04.881610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.881580    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca" pod "994fd1b9-4e15-49a3-9446-5630966e7517" (UID: "994fd1b9-4e15-49a3-9446-5630966e7517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:04.882006 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.881978    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h" pod "994fd1b9-4e15-49a3-9446-5630966e7517" (UID: "994fd1b9-4e15-49a3-9446-5630966e7517"). InnerVolumeSpecName "kube-api-access-qdk6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:04.982540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.982511    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdk6h\" (UniqueName: \"kubernetes.io/projected/994fd1b9-4e15-49a3-9446-5630966e7517-kube-api-access-qdk6h\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:04.982540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:04.982538    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994fd1b9-4e15-49a3-9446-5630966e7517-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:05.325114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.325080    4214 generic.go:356] "Generic (PLEG): container finished" podID="1954967c-5058-4199-88db-d758095c5a73" containerID="fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1" exitCode=60
Oct 30 03:00:05.325614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.325135    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerDied","Data":"fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"}
Oct 30 03:00:05.325614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.325198    4214 scope.go:122] "RemoveContainer" containerID="632d3b8f48192015974b88460e874ca27030c0953534d3ab0ef006112480b1be"
Oct 30 03:00:05.325614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.325584    4214 scope.go:122] "RemoveContainer" containerID="fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"
Oct 30 03:00:05.325986 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:05.325961    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497_odh-ai-gateway-infra(1954967c-5058-4199-88db-d758095c5a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" podUID="1954967c-5058-4199-88db-d758095c5a73"
Oct 30 03:00:05.327484 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.327455    4214 generic.go:356] "Generic (PLEG): container finished" podID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerID="63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30" exitCode=60
Oct 30 03:00:05.327588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.327548    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerDied","Data":"63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"}
Oct 30 03:00:05.327805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.327788    4214 scope.go:122] "RemoveContainer" containerID="63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"
Oct 30 03:00:05.328166 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:05.328113    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-l9t27_odh-ai-gateway-infra(3e956e08-0d5d-485e-b4d6-70f9b6aa251c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c"
Oct 30 03:00:05.329114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.329099    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"
Oct 30 03:00:05.329220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.329128    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f" event={"ID":"994fd1b9-4e15-49a3-9446-5630966e7517","Type":"ContainerDied","Data":"697d32b84560f84b38a64ce5fcb187802536087ed6f6fda0bf0b8d9791646282"}
Oct 30 03:00:05.329220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.329166    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697d32b84560f84b38a64ce5fcb187802536087ed6f6fda0bf0b8d9791646282"
Oct 30 03:00:05.331381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.331290    4214 generic.go:356] "Generic (PLEG): container finished" podID="771039f7-3134-4c81-af26-38a52d5bb76f" containerID="85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd" exitCode=60
Oct 30 03:00:05.331490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.331399    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerDied","Data":"85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"}
Oct 30 03:00:05.331662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.331641    4214 scope.go:122] "RemoveContainer" containerID="85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"
Oct 30 03:00:05.331920 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:05.331896    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb_odh-ai-gateway-infra(771039f7-3134-4c81-af26-38a52d5bb76f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" podUID="771039f7-3134-4c81-af26-38a52d5bb76f"
Oct 30 03:00:05.333511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.333491    4214 generic.go:356] "Generic (PLEG): container finished" podID="7a080b24-8407-45fe-8eff-86d30034ee70" containerID="b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6" exitCode=60
Oct 30 03:00:05.333606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.333565    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerDied","Data":"b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"}
Oct 30 03:00:05.333740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.333723    4214 scope.go:122] "RemoveContainer" containerID="b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"
Oct 30 03:00:05.333959 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:05.333940    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8_odh-ai-gateway-infra(7a080b24-8407-45fe-8eff-86d30034ee70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" podUID="7a080b24-8407-45fe-8eff-86d30034ee70"
Oct 30 03:00:05.345257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.341488    4214 scope.go:122] "RemoveContainer" containerID="3b5d79fd7fbe5dedbab7401e56555da5524c98f1a2c90fb602bc500b35bf18f6"
Oct 30 03:00:05.361447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.361425    4214 scope.go:122] "RemoveContainer" containerID="9b1404519e1875778aa1eea4376bcb4f482ec9a19a3eaa91e53c105786e526d5"
Oct 30 03:00:05.382834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:05.382813    4214 scope.go:122] "RemoveContainer" containerID="68e2dd364f56068ebe49ec2d32fb0788b5ef73d2754a88d49bbb321a93432980"
Oct 30 03:00:06.098841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.098805    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:06.099653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.099634    4214 cpu_manager.go:414] "containerMap: removing container" podUID="994fd1b9-4e15-49a3-9446-5630966e7517" containerName="revoke-keys"
Oct 30 03:00:06.099653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.099654    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994fd1b9-4e15-49a3-9446-5630966e7517" containerName="revoke-keys"
Oct 30 03:00:06.099801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.099793    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="994fd1b9-4e15-49a3-9446-5630966e7517" containerName="revoke-keys"
Oct 30 03:00:06.105138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.105119    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.128956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.128928    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:06.133128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.133101    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-adca906b-serving-cert\""
Oct 30 03:00:06.201006 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.200968    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvcp\" (UniqueName: \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.201176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.201071    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.201308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.201276    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.201539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.201515    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.303529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.303498    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvcp\" (UniqueName: \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.303685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.303560    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.303685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.303599    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.303685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.303655    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.304316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.304293    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.305965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.305938    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.306099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.306078    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.322859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.322838    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvcp\" (UniqueName: \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\") pod \"maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.426305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.422714    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:06.444252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.439527    4214 scope.go:122] "RemoveContainer" containerID="85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"
Oct 30 03:00:06.444252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.439736    4214 scope.go:122] "RemoveContainer" containerID="b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"
Oct 30 03:00:06.444252 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:06.440162    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8_odh-ai-gateway-infra(7a080b24-8407-45fe-8eff-86d30034ee70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" podUID="7a080b24-8407-45fe-8eff-86d30034ee70"
Oct 30 03:00:06.448252 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:06.447355    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb_odh-ai-gateway-infra(771039f7-3134-4c81-af26-38a52d5bb76f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" podUID="771039f7-3134-4c81-af26-38a52d5bb76f"
Oct 30 03:00:06.547306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.542909    4214 scope.go:122] "RemoveContainer" containerID="fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"
Oct 30 03:00:06.547306 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:06.543382    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497_odh-ai-gateway-infra(1954967c-5058-4199-88db-d758095c5a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" podUID="1954967c-5058-4199-88db-d758095c5a73"
Oct 30 03:00:06.555550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.555524    4214 scope.go:122] "RemoveContainer" containerID="63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"
Oct 30 03:00:06.555886 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:06.555863    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-l9t27_odh-ai-gateway-infra(3e956e08-0d5d-485e-b4d6-70f9b6aa251c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c"
Oct 30 03:00:06.847166 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:06.847131    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f487a2_f400_4777_94fe_0ebbe97fa719.slice/crio-41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf WatchSource:0}: Error finding container 41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf: Status 404 returned error can't find the container with id 41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf
Oct 30 03:00:06.850288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:06.850260    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:07.593294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:07.593214    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" event={"ID":"15f487a2-f400-4777-94fe-0ebbe97fa719","Type":"ContainerStarted","Data":"be06d23b8d5e9788e88c35430c32b8dee0e5a9a2377e813e27c83a3ada243465"}
Oct 30 03:00:07.593822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:07.593325    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" event={"ID":"15f487a2-f400-4777-94fe-0ebbe97fa719","Type":"ContainerStarted","Data":"41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf"}
Oct 30 03:00:07.597185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:07.596592    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:07.635584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:07.635531    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" podStartSLOduration=1.232088494 podStartE2EDuration="1.635510632s" totalImagesPullingTime="403.422138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:07.633759726 +0000 UTC m=+1146.707930523" watchObservedRunningTime="2026-10-30 03:00:07.635510632 +0000 UTC m=+1146.709681425"
Oct 30 03:00:08.324435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:08.324402    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 03:00:09.185252 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:09.184961    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:00:10.726126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:10.726092    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:00:10.726711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:10.726142    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:11.031715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:11.031631    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"]
Oct 30 03:00:11.045194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:11.045165    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ba5cb2-lzx44"]
Oct 30 03:00:11.071708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:11.071671    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd046002-da02-43d8-b342-473452ebea47" path="/var/lib/kubelet/pods/bd046002-da02-43d8-b342-473452ebea47/volumes"
Oct 30 03:00:11.561082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:11.561048    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:14.594508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.594467    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:00:14.605705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.602720    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.624357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.624323    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:00:14.655047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655013    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655079    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655289    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655442    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bks\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655490    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655648    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655858    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.655975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.655955    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.656052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.656024    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757336    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bks\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757378    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757445    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757569    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757622    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757665    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757784    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757831    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757850    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.757960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.757927    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.758151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.758077    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.758151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.758122    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.758303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.758283    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.758358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.758333    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.759773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.759754    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.760024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.760006    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.798804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.798775    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:14.799525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.799491    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.799659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.799608    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bks\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:14.934223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:14.934132    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:15.293150 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:15.293121    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82cfe94_cdc1_45a8_b793_f933637b8f39.slice/crio-b56747c469ceb9abab7f85d92918c283f48205676ba3b9531562b24115831f4b WatchSource:0}: Error finding container b56747c469ceb9abab7f85d92918c283f48205676ba3b9531562b24115831f4b: Status 404 returned error can't find the container with id b56747c469ceb9abab7f85d92918c283f48205676ba3b9531562b24115831f4b
Oct 30 03:00:15.294836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.294811    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:00:15.296377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.296343    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.296493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.296410    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.296493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.296439    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:15.860645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.860607    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" event={"ID":"f82cfe94-cdc1-45a8-b793-f933637b8f39","Type":"ContainerStarted","Data":"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"}
Oct 30 03:00:15.861151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.860674    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" event={"ID":"f82cfe94-cdc1-45a8-b793-f933637b8f39","Type":"ContainerStarted","Data":"b56747c469ceb9abab7f85d92918c283f48205676ba3b9531562b24115831f4b"}
Oct 30 03:00:15.908593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.908537    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podStartSLOduration=1.9085251109999999 podStartE2EDuration="1.908525111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:15.904061456 +0000 UTC m=+1154.978232249" watchObservedRunningTime="2026-10-30 03:00:15.908525111 +0000 UTC m=+1154.982695903"
Oct 30 03:00:15.935047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:15.935014    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:16.949251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:16.942436    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:16.949251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:16.942493    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:17.070371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.070335    4214 scope.go:122] "RemoveContainer" containerID="fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"
Oct 30 03:00:17.074320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.071357    4214 scope.go:122] "RemoveContainer" containerID="63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"
Oct 30 03:00:17.074320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.071867    4214 scope.go:122] "RemoveContainer" containerID="85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"
Oct 30 03:00:17.885977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.885939    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerStarted","Data":"9a4c617ee0414ef61df8b9420e8e224c92f111b80d52998cb92e876ef6339cf0"}
Oct 30 03:00:17.888286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.888256    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerStarted","Data":"f58f58099f2a2b2b8fe361793ee6031956505330ce3946f95859d368b99e3de7"}
Oct 30 03:00:17.890547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.890519    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerStarted","Data":"341650ebd25c5fa23a28cb7f2149c9d1de64e6d40b5305627cb79924d83a8898"}
Oct 30 03:00:17.952252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.945784    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:17.952252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:17.945834    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:18.903169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.903140    4214 generic.go:356] "Generic (PLEG): container finished" podID="1954967c-5058-4199-88db-d758095c5a73" containerID="9a4c617ee0414ef61df8b9420e8e224c92f111b80d52998cb92e876ef6339cf0" exitCode=60
Oct 30 03:00:18.903361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.903214    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerDied","Data":"9a4c617ee0414ef61df8b9420e8e224c92f111b80d52998cb92e876ef6339cf0"}
Oct 30 03:00:18.903361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.903306    4214 scope.go:122] "RemoveContainer" containerID="fb5e25b1514cdb33671a6da3bc031c93693cab87c0d2d6f99311a98c869000c1"
Oct 30 03:00:18.903720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.903696    4214 scope.go:122] "RemoveContainer" containerID="9a4c617ee0414ef61df8b9420e8e224c92f111b80d52998cb92e876ef6339cf0"
Oct 30 03:00:18.904070 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:18.904045    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497_odh-ai-gateway-infra(1954967c-5058-4199-88db-d758095c5a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" podUID="1954967c-5058-4199-88db-d758095c5a73"
Oct 30 03:00:18.905495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.905476    4214 generic.go:356] "Generic (PLEG): container finished" podID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerID="f58f58099f2a2b2b8fe361793ee6031956505330ce3946f95859d368b99e3de7" exitCode=60
Oct 30 03:00:18.905582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.905549    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerDied","Data":"f58f58099f2a2b2b8fe361793ee6031956505330ce3946f95859d368b99e3de7"}
Oct 30 03:00:18.905918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.905893    4214 scope.go:122] "RemoveContainer" containerID="f58f58099f2a2b2b8fe361793ee6031956505330ce3946f95859d368b99e3de7"
Oct 30 03:00:18.906194 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:18.906175    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-l9t27_odh-ai-gateway-infra(3e956e08-0d5d-485e-b4d6-70f9b6aa251c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c"
Oct 30 03:00:18.907592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.907572    4214 generic.go:356] "Generic (PLEG): container finished" podID="771039f7-3134-4c81-af26-38a52d5bb76f" containerID="341650ebd25c5fa23a28cb7f2149c9d1de64e6d40b5305627cb79924d83a8898" exitCode=60
Oct 30 03:00:18.907697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.907640    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerDied","Data":"341650ebd25c5fa23a28cb7f2149c9d1de64e6d40b5305627cb79924d83a8898"}
Oct 30 03:00:18.907967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.907951    4214 scope.go:122] "RemoveContainer" containerID="341650ebd25c5fa23a28cb7f2149c9d1de64e6d40b5305627cb79924d83a8898"
Oct 30 03:00:18.908201 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:18.908183    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb_odh-ai-gateway-infra(771039f7-3134-4c81-af26-38a52d5bb76f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" podUID="771039f7-3134-4c81-af26-38a52d5bb76f"
Oct 30 03:00:18.922197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.922181    4214 scope.go:122] "RemoveContainer" containerID="63d2e54ac0300eec8e9916976256f4d69d91ea7a48fbca1d870b636163a71d30"
Oct 30 03:00:18.945257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.942610    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:18.945257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.942660    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:18.975581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.975543    4214 scope.go:122] "RemoveContainer" containerID="85a1b8419adb1b2135cca4c916983243ab6baf5e6924a212256e25a4492a38fd"
Oct 30 03:00:18.986637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:18.986494    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:19.016098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.016065    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:19.041254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.035693    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:19.083742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.083708    4214 scope.go:122] "RemoveContainer" containerID="b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"
Oct 30 03:00:19.921267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.921237    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerStarted","Data":"d60a57c9818677e733e675c4133b9dbc8d16e273de2b37de8d9da08075aee72d"}
Oct 30 03:00:19.958253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.938658    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:19.958253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:19.938716    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:20.279779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.262428    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:20.326318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.326209    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\") pod \"771039f7-3134-4c81-af26-38a52d5bb76f\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") "
Oct 30 03:00:20.326457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.326415    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\" (UniqueName: \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\") pod \"771039f7-3134-4c81-af26-38a52d5bb76f\" (UID: \"771039f7-3134-4c81-af26-38a52d5bb76f\") "
Oct 30 03:00:20.327827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.327791    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca" pod "771039f7-3134-4c81-af26-38a52d5bb76f" (UID: "771039f7-3134-4c81-af26-38a52d5bb76f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:20.333719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.333695    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf" pod "771039f7-3134-4c81-af26-38a52d5bb76f" (UID: "771039f7-3134-4c81-af26-38a52d5bb76f"). InnerVolumeSpecName "kube-api-access-d62kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:20.391615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.391588    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:20.396356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.396336    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:20.428288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.428265    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\" (UniqueName: \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\") pod \"1954967c-5058-4199-88db-d758095c5a73\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") "
Oct 30 03:00:20.428458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.428446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\" (UniqueName: \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\") pod \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") "
Oct 30 03:00:20.428553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.428539    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\") pod \"1954967c-5058-4199-88db-d758095c5a73\" (UID: \"1954967c-5058-4199-88db-d758095c5a73\") "
Oct 30 03:00:20.428731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.428716    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\") pod \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\" (UID: \"3e956e08-0d5d-485e-b4d6-70f9b6aa251c\") "
Oct 30 03:00:20.429089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429059    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca" pod "3e956e08-0d5d-485e-b4d6-70f9b6aa251c" (UID: "3e956e08-0d5d-485e-b4d6-70f9b6aa251c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:20.429155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429081    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca" pod "1954967c-5058-4199-88db-d758095c5a73" (UID: "1954967c-5058-4199-88db-d758095c5a73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:20.429644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429628    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.429754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429744    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771039f7-3134-4c81-af26-38a52d5bb76f-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.429808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429801    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1954967c-5058-4199-88db-d758095c5a73-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.429855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.429848    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62kf\" (UniqueName: \"kubernetes.io/projected/771039f7-3134-4c81-af26-38a52d5bb76f-kube-api-access-d62kf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.430423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.430403    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f" pod "1954967c-5058-4199-88db-d758095c5a73" (UID: "1954967c-5058-4199-88db-d758095c5a73"). InnerVolumeSpecName "kube-api-access-gfc9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:20.430895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.430876    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp" pod "3e956e08-0d5d-485e-b4d6-70f9b6aa251c" (UID: "3e956e08-0d5d-485e-b4d6-70f9b6aa251c"). InnerVolumeSpecName "kube-api-access-lnfsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:20.531197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.531128    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfc9f\" (UniqueName: \"kubernetes.io/projected/1954967c-5058-4199-88db-d758095c5a73-kube-api-access-gfc9f\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.531371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.531357    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfsp\" (UniqueName: \"kubernetes.io/projected/3e956e08-0d5d-485e-b4d6-70f9b6aa251c-kube-api-access-lnfsp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:20.945255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.944993    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:20.945255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.945052    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:20.988266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.982585    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497" event={"ID":"1954967c-5058-4199-88db-d758095c5a73","Type":"ContainerDied","Data":"859142bf678b88bd010918f29875ab46735f06783e718054d03ca9afef34ce76"}
Oct 30 03:00:20.988266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.982684    4214 scope.go:122] "RemoveContainer" containerID="9a4c617ee0414ef61df8b9420e8e224c92f111b80d52998cb92e876ef6339cf0"
Oct 30 03:00:20.988266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:20.982800    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"
Oct 30 03:00:21.012590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.012555    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"
Oct 30 03:00:21.012847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.012818    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27" event={"ID":"3e956e08-0d5d-485e-b4d6-70f9b6aa251c","Type":"ContainerDied","Data":"45c68c2264f62aaf5297e93a61d5a6a9ebc3862c6c412259ed688b3d1114691f"}
Oct 30 03:00:21.032462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.032434    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"
Oct 30 03:00:21.033483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.033416    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb" event={"ID":"771039f7-3134-4c81-af26-38a52d5bb76f","Type":"ContainerDied","Data":"96f2fa35517fb6c4f08ac72d00e58c44b2ed722f1ff178ea48d1e41f68d7a549"}
Oct 30 03:00:21.103706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.103086    4214 generic.go:356] "Generic (PLEG): container finished" podID="7a080b24-8407-45fe-8eff-86d30034ee70" containerID="d60a57c9818677e733e675c4133b9dbc8d16e273de2b37de8d9da08075aee72d" exitCode=60
Oct 30 03:00:21.104075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.104040    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerDied","Data":"d60a57c9818677e733e675c4133b9dbc8d16e273de2b37de8d9da08075aee72d"}
Oct 30 03:00:21.104541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.104517    4214 scope.go:122] "RemoveContainer" containerID="d60a57c9818677e733e675c4133b9dbc8d16e273de2b37de8d9da08075aee72d"
Oct 30 03:00:21.105046 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:21.105011    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8_odh-ai-gateway-infra(7a080b24-8407-45fe-8eff-86d30034ee70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" podUID="7a080b24-8407-45fe-8eff-86d30034ee70"
Oct 30 03:00:21.113852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.113834    4214 scope.go:122] "RemoveContainer" containerID="f58f58099f2a2b2b8fe361793ee6031956505330ce3946f95859d368b99e3de7"
Oct 30 03:00:21.127097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.127069    4214 scope.go:122] "RemoveContainer" containerID="341650ebd25c5fa23a28cb7f2149c9d1de64e6d40b5305627cb79924d83a8898"
Oct 30 03:00:21.172657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.172634    4214 scope.go:122] "RemoveContainer" containerID="b4c5f7578d037917953d0d1584d14f8989d4cc230da049a99c01bcbd555d35b6"
Oct 30 03:00:21.176527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.176501    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:21.182637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.182335    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-l9t27"]
Oct 30 03:00:21.201864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.201805    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:21.214769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.214725    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888820-8g497"]
Oct 30 03:00:21.222436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.222411    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:21.304609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.304211    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:21.310558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.310523    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-32bad3-29888820-b6bnb"]
Oct 30 03:00:21.944266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.938652    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:21.944266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:21.938713    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:22.349624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.349593    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:22.399564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.399532    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\" (UniqueName: \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\") pod \"7a080b24-8407-45fe-8eff-86d30034ee70\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") "
Oct 30 03:00:22.399755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.399719    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\") pod \"7a080b24-8407-45fe-8eff-86d30034ee70\" (UID: \"7a080b24-8407-45fe-8eff-86d30034ee70\") "
Oct 30 03:00:22.400132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.400089    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca" pod "7a080b24-8407-45fe-8eff-86d30034ee70" (UID: "7a080b24-8407-45fe-8eff-86d30034ee70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:22.402113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.402086    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a080b24-8407-45fe-8eff-86d30034ee70-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:22.406442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.406412    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l" pod "7a080b24-8407-45fe-8eff-86d30034ee70" (UID: "7a080b24-8407-45fe-8eff-86d30034ee70"). InnerVolumeSpecName "kube-api-access-fxq2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:22.503405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.503331    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxq2l\" (UniqueName: \"kubernetes.io/projected/7a080b24-8407-45fe-8eff-86d30034ee70-kube-api-access-fxq2l\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:22.934545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.934509    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 03:00:22.934742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:22.934570    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 03:00:23.070215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.070174    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1954967c-5058-4199-88db-d758095c5a73" path="/var/lib/kubelet/pods/1954967c-5058-4199-88db-d758095c5a73/volumes"
Oct 30 03:00:23.070742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.070719    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" path="/var/lib/kubelet/pods/3e956e08-0d5d-485e-b4d6-70f9b6aa251c/volumes"
Oct 30 03:00:23.071215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.071195    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" path="/var/lib/kubelet/pods/771039f7-3134-4c81-af26-38a52d5bb76f/volumes"
Oct 30 03:00:23.131674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.131640    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"
Oct 30 03:00:23.131674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.131656    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8" event={"ID":"7a080b24-8407-45fe-8eff-86d30034ee70","Type":"ContainerDied","Data":"5265fd52670463821085fb6a8449c9679598cedb2e3ec645e43e379d33513636"}
Oct 30 03:00:23.131875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.131739    4214 scope.go:122] "RemoveContainer" containerID="d60a57c9818677e733e675c4133b9dbc8d16e273de2b37de8d9da08075aee72d"
Oct 30 03:00:23.190303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.190197    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:23.201968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.201940    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888820-stbq8"]
Oct 30 03:00:23.955963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.955929    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:23.956799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.956768    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:23.958178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:23.958155    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:00:24.033924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:24.033892    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"]
Oct 30 03:00:24.042545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:24.042521    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c43ecdd1-7r4zz"]
Oct 30 03:00:25.187251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:25.182939    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439243d3-1673-4857-9edd-461f0d2c176a" path="/var/lib/kubelet/pods/439243d3-1673-4857-9edd-461f0d2c176a/volumes"
Oct 30 03:00:25.187251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:25.183671    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" path="/var/lib/kubelet/pods/7a080b24-8407-45fe-8eff-86d30034ee70/volumes"
Oct 30 03:00:28.094254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:28.093596    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"]
Oct 30 03:00:28.349310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:28.345566    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49690c32-zth8c"]
Oct 30 03:00:29.244412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:29.244365    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9bbf74-0972-42c4-b806-5c3d4cb24299" path="/var/lib/kubelet/pods/1e9bbf74-0972-42c4-b806-5c3d4cb24299/volumes"
Oct 30 03:00:29.245536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:29.245511    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"]
Oct 30 03:00:29.245713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:29.245697    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c387fc0-swsjh"]
Oct 30 03:00:29.707267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:29.703336    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:29.818249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:29.807415    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.721361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:30.721295    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.721960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:30.721626    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:00:30.845253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:30.826829    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:30.845253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:30.827406    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:00:31.223253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.167326    4214 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:00:31.223253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.167388    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:31.349264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.304914    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:31.486297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.483337    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faa5401-cd45-4be5-b950-1578a1cbdd4b" path="/var/lib/kubelet/pods/5faa5401-cd45-4be5-b950-1578a1cbdd4b/volumes"
Oct 30 03:00:31.737303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.736474    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:31.737303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.736625    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:31.830528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.830423    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:31.908683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:31.907752    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:00:32.032995 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:32.023459    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:00:32.045437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:32.044157    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:00:37.624517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:37.624434    4214 patch_prober.go:29] interesting pod/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:00:37.624991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:37.624553    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:39.198335 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:39.198292    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:00:40.053252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:40.045543    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"]
Oct 30 03:00:40.076528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:40.076492    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89089d14-jts7h"]
Oct 30 03:00:41.087746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.087712    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ba7dc3-2eaf-475e-96a9-975b85c33fa3" path="/var/lib/kubelet/pods/d1ba7dc3-2eaf-475e-96a9-975b85c33fa3/volumes"
Oct 30 03:00:41.190624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.190541    4214 generic.go:356] "Generic (PLEG): container finished" podID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerID="00a1b2bf9f19e2e9b6fa58d333342c45a0f0f3adc14a8d5916c2d43b27db078d" exitCode=0
Oct 30 03:00:41.190624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.190612    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerDied","Data":"00a1b2bf9f19e2e9b6fa58d333342c45a0f0f3adc14a8d5916c2d43b27db078d"}
Oct 30 03:00:41.191103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.191083    4214 scope.go:122] "RemoveContainer" containerID="00a1b2bf9f19e2e9b6fa58d333342c45a0f0f3adc14a8d5916c2d43b27db078d"
Oct 30 03:00:41.192789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.192764    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:41.192879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.192840    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:41.192879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:41.192874    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:42.242250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:42.239179    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerStarted","Data":"1ee63eb3d6f3bfbb716e12c0985f08daa4e0dceeba2edb6181920c8a08e1d9ec"}
Oct 30 03:00:42.980253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:42.978469    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 03:00:42.980253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:42.978559    4214 patch_prober.go:29] interesting pod/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:00:42.980253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:42.978616    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:00:43.102769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:43.099451    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"]
Oct 30 03:00:43.103652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:43.103613    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921a8a19-g4h84"]
Oct 30 03:00:43.789255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:43.778432    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 03:00:43.797627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:43.797586    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" podUID="75b933e0-e300-48f4-8e0c-64290e7c74c9" containerName="maas-api" containerID="cri-o://082ae025149388a267ddd5c179e93b327839bb8c7e8ea34a5aae57eb178bb86e" gracePeriod=30
Oct 30 03:00:44.011264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:44.007412    4214 patch_prober.go:29] interesting pod/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:00:44.011264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:44.007474    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:00:44.510255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:44.491300    4214 generic.go:356] "Generic (PLEG): container finished" podID="75b933e0-e300-48f4-8e0c-64290e7c74c9" containerID="082ae025149388a267ddd5c179e93b327839bb8c7e8ea34a5aae57eb178bb86e" exitCode=0
Oct 30 03:00:44.510255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:44.491580    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" event={"ID":"75b933e0-e300-48f4-8e0c-64290e7c74c9","Type":"ContainerDied","Data":"082ae025149388a267ddd5c179e93b327839bb8c7e8ea34a5aae57eb178bb86e"}
Oct 30 03:00:45.054456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.054419    4214 patch_prober.go:29] interesting pod/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:00:45.120639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.094017    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:00:45.120639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.068749    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:45.315300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.311751    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40afb98d-2fb4-4604-87ce-cc59f9143221" path="/var/lib/kubelet/pods/40afb98d-2fb4-4604-87ce-cc59f9143221/volumes"
Oct 30 03:00:45.330355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.330310    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:45.350588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.350542    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:45.380255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.361872    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 03:00:45.380255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.370725    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy" containerID="cri-o://1ee63eb3d6f3bfbb716e12c0985f08daa4e0dceeba2edb6181920c8a08e1d9ec" gracePeriod=30
Oct 30 03:00:45.752275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.752154    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68" event={"ID":"75b933e0-e300-48f4-8e0c-64290e7c74c9","Type":"ContainerDied","Data":"b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592"}
Oct 30 03:00:45.761614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.761583    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b52d6f938ccbdcfdae760ff5bfd4d57af06a5f5e30fb02520e544108e2851592"
Oct 30 03:00:45.766467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.766438    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 03:00:45.874859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.874816    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\") pod \"75b933e0-e300-48f4-8e0c-64290e7c74c9\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") "
Oct 30 03:00:45.875353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.875327    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\") pod \"75b933e0-e300-48f4-8e0c-64290e7c74c9\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") "
Oct 30 03:00:45.875602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.875586    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") pod \"75b933e0-e300-48f4-8e0c-64290e7c74c9\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") "
Oct 30 03:00:45.875769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.875751    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\" (UniqueName: \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\") pod \"75b933e0-e300-48f4-8e0c-64290e7c74c9\" (UID: \"75b933e0-e300-48f4-8e0c-64290e7c74c9\") "
Oct 30 03:00:45.882635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.882510    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca" pod "75b933e0-e300-48f4-8e0c-64290e7c74c9" (UID: "75b933e0-e300-48f4-8e0c-64290e7c74c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:45.925881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.925844    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb" pod "75b933e0-e300-48f4-8e0c-64290e7c74c9" (UID: "75b933e0-e300-48f4-8e0c-64290e7c74c9"). InnerVolumeSpecName "kube-api-access-686bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:45.935394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.927422    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs" pod "75b933e0-e300-48f4-8e0c-64290e7c74c9" (UID: "75b933e0-e300-48f4-8e0c-64290e7c74c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:45.935827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.935799    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls" pod "75b933e0-e300-48f4-8e0c-64290e7c74c9" (UID: "75b933e0-e300-48f4-8e0c-64290e7c74c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:46.002365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:45.980074    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b933e0-e300-48f4-8e0c-64290e7c74c9-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:46.081527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.081497    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:46.082105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.082087    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b933e0-e300-48f4-8e0c-64290e7c74c9-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:46.082214 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.082201    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686bb\" (UniqueName: \"kubernetes.io/projected/75b933e0-e300-48f4-8e0c-64290e7c74c9-kube-api-access-686bb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:46.133254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.096674    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:46.133254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.097269    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:00:46.394410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.332526    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:00:46.394410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.333384    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:00:46.417284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.417253    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:00:46.431095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.431035    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:00:46.893267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:46.891463    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"
Oct 30 03:00:47.212702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.212470    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213564    4214 cpu_manager.go:414] "containerMap: removing container" podUID="75b933e0-e300-48f4-8e0c-64290e7c74c9" containerName="maas-api"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213586    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b933e0-e300-48f4-8e0c-64290e7c74c9" containerName="maas-api"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213604    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213612    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213647    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213655    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213702    4214 cpu_manager.go:414] "containerMap: removing container" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213713    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213728    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213738    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213754    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213765    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213777    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.213782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213788    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213806    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213816    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213844    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213854    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213874    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213884    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213907    4214 cpu_manager.go:414] "containerMap: removing container" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.213916    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214111    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b933e0-e300-48f4-8e0c-64290e7c74c9" containerName="maas-api"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214125    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214138    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214150    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214158    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214171    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214180    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214192    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e956e08-0d5d-485e-b4d6-70f9b6aa251c" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214201    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214210    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a080b24-8407-45fe-8eff-86d30034ee70" containerName="cleanup"
Oct 30 03:00:47.214714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.214222    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:47.219593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.219563    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:00:47.220258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.220218    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 03:00:47.220367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.220334    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.228047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.228005    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2b0506d-588bb469bb-6cm68"]
Oct 30 03:00:47.239403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.239381    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9ca6fbc2-gw-openshift-default-dockercfg-25pct\""
Oct 30 03:00:47.344718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.344678    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.345107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.345080    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.345729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.345707    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.346540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.346519    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zt5\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.346731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.346712    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.361961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.361931    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.362336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.362308    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.362680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.362659    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.362832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.362815    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.470380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.470254    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.470696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.470670    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.470934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.470914    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471116    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471263    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471403    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471551    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471774    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.471942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.471928    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zt5\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.473092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.473071    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.488740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.473718    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.488938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.474796    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.489059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.475025    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.489162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.485404    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.513927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.505895    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.513927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.506438    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.513927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.508361    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.515973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.515949    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zt5\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\") pod \"e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") " pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.573406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.569525    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:47.959204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.959166    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:00:47.975603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.975249    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:47.975603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.975339    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:47.975603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:47.975383    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:48.177252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:48.168790    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:48.177252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:48.169074    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" podUID="23ef435d-2d75-4dbd-8159-f7b5537576e0" containerName="maas-api" containerID="cri-o://ce431cb00716ba4d32efdc37e00862feace98790383667fc4b207921c8af70aa" gracePeriod=30
Oct 30 03:00:49.455332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.455148    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:00:49.484963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.484924    4214 generic.go:356] "Generic (PLEG): container finished" podID="23ef435d-2d75-4dbd-8159-f7b5537576e0" containerID="ce431cb00716ba4d32efdc37e00862feace98790383667fc4b207921c8af70aa" exitCode=0
Oct 30 03:00:49.520281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.493671    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b933e0-e300-48f4-8e0c-64290e7c74c9" path="/var/lib/kubelet/pods/75b933e0-e300-48f4-8e0c-64290e7c74c9/volumes"
Oct 30 03:00:49.521312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.521278    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" event={"ID":"23ef435d-2d75-4dbd-8159-f7b5537576e0","Type":"ContainerDied","Data":"ce431cb00716ba4d32efdc37e00862feace98790383667fc4b207921c8af70aa"}
Oct 30 03:00:49.718190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.718098    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerStarted","Data":"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"}
Oct 30 03:00:49.718495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.718457    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerStarted","Data":"057c88610dbff8752892b9d558b09955c4232c6c252b897fa63525f54c0b19b5"}
Oct 30 03:00:49.938268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.911023    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:49.938878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:49.938830    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:50.608929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.608890    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:50.609687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.609635    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:50.662511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.662458    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:00:50.750023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.749921    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:00:50.870252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.866700    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 03:00:50.870252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.866980    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerName="istio-proxy" containerID="cri-o://f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41" gracePeriod=30
Oct 30 03:00:50.992527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.952355    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:50.992527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:50.969175    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:00:51.373721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.373571    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podStartSLOduration=5.373549298 podStartE2EDuration="5.373549298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:50.166930593 +0000 UTC m=+1189.241101387" watchObservedRunningTime="2026-10-30 03:00:51.373549298 +0000 UTC m=+1190.447720083"
Oct 30 03:00:51.523556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.523462    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:51.628254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.624037    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:51.715255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.704355    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:51.715255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.704456    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 03:00:51.715255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.704505    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:51.715255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.704553    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:00:51.715255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.704572    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:51.865255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.858205    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:51.865255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.861812    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\") pod \"23ef435d-2d75-4dbd-8159-f7b5537576e0\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") "
Oct 30 03:00:51.865255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.861966    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\") pod \"23ef435d-2d75-4dbd-8159-f7b5537576e0\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") "
Oct 30 03:00:51.865255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.862048    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\") pod \"23ef435d-2d75-4dbd-8159-f7b5537576e0\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") "
Oct 30 03:00:51.865255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.862219    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\" (UniqueName: \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\") pod \"23ef435d-2d75-4dbd-8159-f7b5537576e0\" (UID: \"23ef435d-2d75-4dbd-8159-f7b5537576e0\") "
Oct 30 03:00:51.916583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.916530    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 03:00:51.960482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.960434    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:51.965185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.965147    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs" pod "23ef435d-2d75-4dbd-8159-f7b5537576e0" (UID: "23ef435d-2d75-4dbd-8159-f7b5537576e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:51.979270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:51.971731    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls" pod "23ef435d-2d75-4dbd-8159-f7b5537576e0" (UID: "23ef435d-2d75-4dbd-8159-f7b5537576e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:52.088784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.088747    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:52.105973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.088993    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ef435d-2d75-4dbd-8159-f7b5537576e0-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:52.120773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.107703    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:00:52.151297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.133812    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca" pod "23ef435d-2d75-4dbd-8159-f7b5537576e0" (UID: "23ef435d-2d75-4dbd-8159-f7b5537576e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:52.151297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.133975    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x" pod "23ef435d-2d75-4dbd-8159-f7b5537576e0" (UID: "23ef435d-2d75-4dbd-8159-f7b5537576e0"). InnerVolumeSpecName "kube-api-access-swp9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:52.232479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.212882    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ef435d-2d75-4dbd-8159-f7b5537576e0-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:52.232479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.212914    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swp9x\" (UniqueName: \"kubernetes.io/projected/23ef435d-2d75-4dbd-8159-f7b5537576e0-kube-api-access-swp9x\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:52.354497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.354454    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:00:52.561668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.561638    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"
Oct 30 03:00:52.568537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.568506    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj" event={"ID":"23ef435d-2d75-4dbd-8159-f7b5537576e0","Type":"ContainerDied","Data":"10cb02b45c879a6f74a65f24051a644f3cf247767cd151bd51c16f2ecd32de09"}
Oct 30 03:00:52.576688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.576667    4214 scope.go:122] "RemoveContainer" containerID="ce431cb00716ba4d32efdc37e00862feace98790383667fc4b207921c8af70aa"
Oct 30 03:00:52.596899 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.596865    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:00:52.654606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.654524    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"]
Oct 30 03:00:52.703414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.703217    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1afaee-gjvgn"]
Oct 30 03:00:52.778087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.778040    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:52.787694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:52.787663    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93fe9de7-bd96b8dbc-l49vj"]
Oct 30 03:00:53.069164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.069122    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ef435d-2d75-4dbd-8159-f7b5537576e0" path="/var/lib/kubelet/pods/23ef435d-2d75-4dbd-8159-f7b5537576e0/volumes"
Oct 30 03:00:53.069712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.069695    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979f96bc-700d-4802-8910-6fa66acb0a32" path="/var/lib/kubelet/pods/979f96bc-700d-4802-8910-6fa66acb0a32/volumes"
Oct 30 03:00:53.745871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.745552    4214 patch_prober.go:29] interesting pod/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:00:53.745871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.745618    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:00:53.784624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.784581    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:00:53.785809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785781    4214 cpu_manager.go:414] "containerMap: removing container" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:53.785912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785812    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:53.785912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785836    4214 cpu_manager.go:414] "containerMap: removing container" podUID="23ef435d-2d75-4dbd-8159-f7b5537576e0" containerName="maas-api"
Oct 30 03:00:53.785912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785844    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ef435d-2d75-4dbd-8159-f7b5537576e0" containerName="maas-api"
Oct 30 03:00:53.785912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785870    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:53.785912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.785877    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:53.786193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.786033    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1954967c-5058-4199-88db-d758095c5a73" containerName="cleanup"
Oct 30 03:00:53.786193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.786056    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ef435d-2d75-4dbd-8159-f7b5537576e0" containerName="maas-api"
Oct 30 03:00:53.786193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.786072    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="771039f7-3134-4c81-af26-38a52d5bb76f" containerName="cleanup"
Oct 30 03:00:53.792277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.792244    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.819655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.819594    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9245d1-openshift-default-dockercfg-884td\""
Oct 30 03:00:53.825752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.825712    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:00:53.910246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910133    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910217    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910370    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910413    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910451    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910500    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910540    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910563    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:53.910682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:53.910580    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043369    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043450    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043535    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043582    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043638    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043708    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043765    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043802    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.043830    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.047298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.044758    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.119249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.118572    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.125928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.119571    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.132880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.132242    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.132880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.132829    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.195298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.192981    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.201382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.201348    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.207023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.206915    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:54.208356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.208098    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerName="maas-api" containerID="cri-o://be06d23b8d5e9788e88c35430c32b8dee0e5a9a2377e813e27c83a3ada243465" gracePeriod=30
Oct 30 03:00:54.330348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.257107    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.330348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.286913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.454254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.452103    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:54.802753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:54.802713    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": dial tcp 10.133.0.133:8443: connect: connection refused"
Oct 30 03:00:55.752276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.752202    4214 generic.go:356] "Generic (PLEG): container finished" podID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerID="be06d23b8d5e9788e88c35430c32b8dee0e5a9a2377e813e27c83a3ada243465" exitCode=0
Oct 30 03:00:55.752481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.752280    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" event={"ID":"15f487a2-f400-4777-94fe-0ebbe97fa719","Type":"ContainerDied","Data":"be06d23b8d5e9788e88c35430c32b8dee0e5a9a2377e813e27c83a3ada243465"}
Oct 30 03:00:55.752481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.752372    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m" event={"ID":"15f487a2-f400-4777-94fe-0ebbe97fa719","Type":"ContainerDied","Data":"41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf"}
Oct 30 03:00:55.752481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.752397    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b58787748f990364ef4b6f97477ad0c9442a8624f9a28cc75fc71a27d9a2bf"
Oct 30 03:00:55.757053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.757031    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:55.772938 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:55.772907    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384248e9_5440_4b3e_851f_4b3a2dcd9f7c.slice/crio-046efcc684d85521f55369373a1061221cd1456b118b0e6139a2815e05deb962 WatchSource:0}: Error finding container 046efcc684d85521f55369373a1061221cd1456b118b0e6139a2815e05deb962: Status 404 returned error can't find the container with id 046efcc684d85521f55369373a1061221cd1456b118b0e6139a2815e05deb962
Oct 30 03:00:55.775871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.775841    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:55.775953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.775904    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:55.775953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.775932    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:00:55.778384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.778360    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:00:55.813874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.811696    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\") pod \"15f487a2-f400-4777-94fe-0ebbe97fa719\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") "
Oct 30 03:00:55.813874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.811783    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\") pod \"15f487a2-f400-4777-94fe-0ebbe97fa719\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") "
Oct 30 03:00:55.813874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.811843    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\") pod \"15f487a2-f400-4777-94fe-0ebbe97fa719\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") "
Oct 30 03:00:55.813874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.812143    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\" (UniqueName: \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\") pod \"15f487a2-f400-4777-94fe-0ebbe97fa719\" (UID: \"15f487a2-f400-4777-94fe-0ebbe97fa719\") "
Oct 30 03:00:55.815304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.815274    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca" pod "15f487a2-f400-4777-94fe-0ebbe97fa719" (UID: "15f487a2-f400-4777-94fe-0ebbe97fa719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:55.817099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.817054    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs" pod "15f487a2-f400-4777-94fe-0ebbe97fa719" (UID: "15f487a2-f400-4777-94fe-0ebbe97fa719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:55.817508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.817485    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp" pod "15f487a2-f400-4777-94fe-0ebbe97fa719" (UID: "15f487a2-f400-4777-94fe-0ebbe97fa719"). InnerVolumeSpecName "kube-api-access-xmvcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:55.820136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.820114    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls" pod "15f487a2-f400-4777-94fe-0ebbe97fa719" (UID: "15f487a2-f400-4777-94fe-0ebbe97fa719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:55.914355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.914279    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvcp\" (UniqueName: \"kubernetes.io/projected/15f487a2-f400-4777-94fe-0ebbe97fa719-kube-api-access-xmvcp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:55.914355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.914312    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15f487a2-f400-4777-94fe-0ebbe97fa719-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:55.914355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.914326    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:55.914355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:55.914336    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15f487a2-f400-4777-94fe-0ebbe97fa719-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:56.768816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.768317    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"
Oct 30 03:00:56.769348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.769314    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerStarted","Data":"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"}
Oct 30 03:00:56.769480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.769377    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerStarted","Data":"046efcc684d85521f55369373a1061221cd1456b118b0e6139a2815e05deb962"}
Oct 30 03:00:56.833693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.833640    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podStartSLOduration=3.8336259200000002 podStartE2EDuration="3.83362592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:56.827671114 +0000 UTC m=+1195.901841909" watchObservedRunningTime="2026-10-30 03:00:56.83362592 +0000 UTC m=+1195.907796712"
Oct 30 03:00:56.918961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.918923    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:56.924192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:56.924165    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-adca906b-f5d7d864d-2mj5m"]
Oct 30 03:00:57.155334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.150687    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" path="/var/lib/kubelet/pods/15f487a2-f400-4777-94fe-0ebbe97fa719/volumes"
Oct 30 03:00:57.457939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.457846    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:57.465598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.465567    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:57.876139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.874396    4214 generic.go:356] "Generic (PLEG): container finished" podID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerID="1ee63eb3d6f3bfbb716e12c0985f08daa4e0dceeba2edb6181920c8a08e1d9ec" exitCode=0
Oct 30 03:00:57.885594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.876644    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerDied","Data":"1ee63eb3d6f3bfbb716e12c0985f08daa4e0dceeba2edb6181920c8a08e1d9ec"}
Oct 30 03:00:57.885594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.876722    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:57.885594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.876755    4214 scope.go:122] "RemoveContainer" containerID="00a1b2bf9f19e2e9b6fa58d333342c45a0f0f3adc14a8d5916c2d43b27db078d"
Oct 30 03:00:57.885594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.879585    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:00:57.939001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:57.938962    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 03:00:58.003272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.003180    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"]
Oct 30 03:00:58.005189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.005160    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.005723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.005705    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.005823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.005812    4214 cpu_manager.go:414] "containerMap: removing container" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerName="maas-api"
Oct 30 03:00:58.005915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.005903    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerName="maas-api"
Oct 30 03:00:58.006053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.006042    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.006132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.006123    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.006571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.006549    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.007948    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f487a2-f400-4777-94fe-0ebbe97fa719" containerName="maas-api"
Oct 30 03:00:58.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.007981    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" containerName="istio-proxy"
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.037694    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.037757    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.037906    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039767    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039857    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039909    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.040099    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.040144    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.051566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.040274    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\") pod \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\" (UID: \"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f\") "
Oct 30 03:00:58.079253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039115    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.038554    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039391    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.039036    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"]
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.089811    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.091016    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.092337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.092130    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.096323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.095428    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.096323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.095456    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.096323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.095488    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.096323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.095504    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.096323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.095519    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.099685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.099647    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "kube-api-access-65p4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:58.101392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.101368    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:00:58.101536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.101415    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:58.101817 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.101797    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy" pod "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" (UID: "3c94de87-f3bc-4bb3-a005-137d9b6cfb9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.105626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.105599    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:58.105730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.105646    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:58.198668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.198624    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.198954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.198931    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzmh\" (UniqueName: \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.199053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.199024    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.199144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.199126    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.199206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.199150    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65p4q\" (UniqueName: \"kubernetes.io/projected/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-kube-api-access-65p4q\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.199206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.199166    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.199206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.199180    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.283633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.283608    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 03:00:58.300180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.300156    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzmh\" (UniqueName: \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.300353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.300214    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.300353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.300307    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.301125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.301100    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.302921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.302895    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.320198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.320177    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzmh\" (UniqueName: \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\") pod \"maas-api-revoke-sub-071e1cb2-rrqjf\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.401185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401148    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401266    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401315    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401488    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401529    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:58.401672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401648    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401720    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401765    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.401985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401916    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.401985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.401949    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.402315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402012    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.402315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402052    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\") pod \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\" (UID: \"6d2f77e9-1653-4d59-9eea-73d261d15e9a\") "
Oct 30 03:00:58.402315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402094    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.402496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402354    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.402828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402802    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.402905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402834    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.402905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402849    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.402905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402863    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.402905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.402875    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.403670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.403650    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:58.403753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.403672    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:00:58.403753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.403685    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "kube-api-access-8wlzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:58.403884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.403867    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo" pod "6d2f77e9-1653-4d59-9eea-73d261d15e9a" (UID: "6d2f77e9-1653-4d59-9eea-73d261d15e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:00:58.416088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.416066    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:00:58.504809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.504339    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wlzq\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-kube-api-access-8wlzq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.504809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.504364    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.504809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.504373    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.504809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.504381    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2f77e9-1653-4d59-9eea-73d261d15e9a-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:00:58.728512 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:00:58.728452    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b04868a_85a5_4035_824f_7c4bef768e74.slice/crio-ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc WatchSource:0}: Error finding container ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc: Status 404 returned error can't find the container with id ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc
Oct 30 03:00:58.729722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.729696    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"]
Oct 30 03:00:58.887907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.887871    4214 generic.go:356] "Generic (PLEG): container finished" podID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerID="f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41" exitCode=0
Oct 30 03:00:58.887907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.887912    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"
Oct 30 03:00:58.888478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.887964    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" event={"ID":"6d2f77e9-1653-4d59-9eea-73d261d15e9a","Type":"ContainerDied","Data":"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"}
Oct 30 03:00:58.888478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.888017    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r" event={"ID":"6d2f77e9-1653-4d59-9eea-73d261d15e9a","Type":"ContainerDied","Data":"14c783eba4fccce26c2099d080c362251501ccbdb46eb256aba8fa8f866d537d"}
Oct 30 03:00:58.888478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.888041    4214 scope.go:122] "RemoveContainer" containerID="f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"
Oct 30 03:00:58.889923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.889897    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf" event={"ID":"6b04868a-85a5-4035-824f-7c4bef768e74","Type":"ContainerStarted","Data":"ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc"}
Oct 30 03:00:58.891804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.891783    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"
Oct 30 03:00:58.891952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.891815    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x" event={"ID":"3c94de87-f3bc-4bb3-a005-137d9b6cfb9f","Type":"ContainerDied","Data":"ea795342bd6ca2441bb32b907394078d2a3f15b90f206d58a58177012e3bb011"}
Oct 30 03:00:58.901111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.901092    4214 scope.go:122] "RemoveContainer" containerID="f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"
Oct 30 03:00:58.901445 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:00:58.901426    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41\": container with ID starting with f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41 not found: ID does not exist" containerID="f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"
Oct 30 03:00:58.901556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.901453    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41"} err="failed to get container status \"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41\": rpc error: code = NotFound desc = could not find container \"f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41\": container with ID starting with f5bdffc2890b28d53974eff5cfe3bdc46c6f003bfdbd5ad30fae9f1d98eeca41 not found: ID does not exist"
Oct 30 03:00:58.901556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.901470    4214 scope.go:122] "RemoveContainer" containerID="1ee63eb3d6f3bfbb716e12c0985f08daa4e0dceeba2edb6181920c8a08e1d9ec"
Oct 30 03:00:58.950244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.950199    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 03:00:58.963990 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.963960    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93fe9de7-openshift-default-b46c945d9-q6v2r"]
Oct 30 03:00:58.990623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.990552    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 03:00:58.995773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:58.995748    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e2b0506d-openshift-default-774994d89c-hjf9x"]
Oct 30 03:00:59.077317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:59.077285    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c94de87-f3bc-4bb3-a005-137d9b6cfb9f" path="/var/lib/kubelet/pods/3c94de87-f3bc-4bb3-a005-137d9b6cfb9f/volumes"
Oct 30 03:00:59.077899 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:59.077883    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" path="/var/lib/kubelet/pods/6d2f77e9-1653-4d59-9eea-73d261d15e9a/volumes"
Oct 30 03:00:59.904994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:59.904959    4214 generic.go:356] "Generic (PLEG): container finished" podID="6b04868a-85a5-4035-824f-7c4bef768e74" containerID="bf220a573158ea181b1f6f67b29cc7ce7fc1ac9bd44b8f54b9df77103bba30bb" exitCode=0
Oct 30 03:00:59.905520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:00:59.905051    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf" event={"ID":"6b04868a-85a5-4035-824f-7c4bef768e74","Type":"ContainerDied","Data":"bf220a573158ea181b1f6f67b29cc7ce7fc1ac9bd44b8f54b9df77103bba30bb"}
Oct 30 03:01:01.174738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.174709    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:01:01.175475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.175455    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerName="istio-proxy"
Oct 30 03:01:01.175475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.175474    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerName="istio-proxy"
Oct 30 03:01:01.175615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.175604    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2f77e9-1653-4d59-9eea-73d261d15e9a" containerName="istio-proxy"
Oct 30 03:01:01.178577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.178555    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.182360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.182338    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:01:01.193766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.193745    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9245d1\""
Oct 30 03:01:01.198031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.198007    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:01:01.278423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.268604    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9245d1-dockercfg-w7xvw\""
Oct 30 03:01:01.316255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.301454    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\") pod \"6b04868a-85a5-4035-824f-7c4bef768e74\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") "
Oct 30 03:01:01.316255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.301514    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\") pod \"6b04868a-85a5-4035-824f-7c4bef768e74\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") "
Oct 30 03:01:01.316255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.301615    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\" (UniqueName: \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\") pod \"6b04868a-85a5-4035-824f-7c4bef768e74\" (UID: \"6b04868a-85a5-4035-824f-7c4bef768e74\") "
Oct 30 03:01:01.316255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.302246    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.316255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.302377    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6s74\" (UniqueName: \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.415252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.401620    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca" pod "6b04868a-85a5-4035-824f-7c4bef768e74" (UID: "6b04868a-85a5-4035-824f-7c4bef768e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:01.446490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.418531    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.446490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.418664    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6s74\" (UniqueName: \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.446490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.418818    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:01.485523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.484866    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.507457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.507413    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh" pod "6b04868a-85a5-4035-824f-7c4bef768e74" (UID: "6b04868a-85a5-4035-824f-7c4bef768e74"). InnerVolumeSpecName "kube-api-access-8pzmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:01.527854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.512372    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca" pod "6b04868a-85a5-4035-824f-7c4bef768e74" (UID: "6b04868a-85a5-4035-824f-7c4bef768e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:01:01.527854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.513602    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6s74\" (UniqueName: \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\") pod \"payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.530791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.528591    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b04868a-85a5-4035-824f-7c4bef768e74-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:01.530791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.528629    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pzmh\" (UniqueName: \"kubernetes.io/projected/6b04868a-85a5-4035-824f-7c4bef768e74-kube-api-access-8pzmh\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:01.640177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.577971    4214 scope.go:122] "RemoveContainer" containerID="6a829376976d66db70cb75972ef0cc46685a19f1af08731a1a711417c423e4de"
Oct 30 03:01:01.809841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.809804    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:01.926967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.926938    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"
Oct 30 03:01:01.926967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.926956    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf" event={"ID":"6b04868a-85a5-4035-824f-7c4bef768e74","Type":"ContainerDied","Data":"ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc"}
Oct 30 03:01:01.927183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:01.927008    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3b4aee5bf125b883a0f9a7cf871e0c9e8957e101a6cb18598c1958c2c2becc"
Oct 30 03:01:02.146798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:02.142367    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:01:02.146798 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:02.144972    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8480052_5a01_473f_bc4a_45a3472e434e.slice/crio-e45148a48d268d64969691001ce4ee71bdb5cdc0d7b53a95b4ff24abd0cdc9ed WatchSource:0}: Error finding container e45148a48d268d64969691001ce4ee71bdb5cdc0d7b53a95b4ff24abd0cdc9ed: Status 404 returned error can't find the container with id e45148a48d268d64969691001ce4ee71bdb5cdc0d7b53a95b4ff24abd0cdc9ed
Oct 30 03:01:02.938172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:02.938128    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" event={"ID":"c8480052-5a01-473f-bc4a-45a3472e434e","Type":"ContainerStarted","Data":"ec16579a17b96b19147285f345e7d1d97e610e3a4ca18cbd1369a2127509475e"}
Oct 30 03:01:02.938585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:02.938224    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" event={"ID":"c8480052-5a01-473f-bc4a-45a3472e434e","Type":"ContainerStarted","Data":"e45148a48d268d64969691001ce4ee71bdb5cdc0d7b53a95b4ff24abd0cdc9ed"}
Oct 30 03:01:02.938585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:02.938418    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:02.983207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:02.983150    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" podStartSLOduration=1.983136872 podStartE2EDuration="1.983136872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:02.977204263 +0000 UTC m=+1202.051375069" watchObservedRunningTime="2026-10-30 03:01:02.983136872 +0000 UTC m=+1202.057307664"
Oct 30 03:01:09.170155 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:01:09.170116    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:01:10.097466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.097425    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:10.099268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.099239    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6b04868a-85a5-4035-824f-7c4bef768e74" containerName="revoke-keys"
Oct 30 03:01:10.099268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.099266    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b04868a-85a5-4035-824f-7c4bef768e74" containerName="revoke-keys"
Oct 30 03:01:10.099653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.099636    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b04868a-85a5-4035-824f-7c4bef768e74" containerName="revoke-keys"
Oct 30 03:01:10.105207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.105177    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.124850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.124822    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9ca6fbc2-serving-cert\""
Oct 30 03:01:10.127528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.127501    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:10.240205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.240164    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.240641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.240371    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.240641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.240561    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.240732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.240692    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbnfb\" (UniqueName: \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.347250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.346643    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.347250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.346871    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.347250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.347125    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbnfb\" (UniqueName: \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.347250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.347205    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.353298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.349625    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.378959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.372911    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.400386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.400308    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.413261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.410072    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbnfb\" (UniqueName: \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\") pod \"maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.429290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.428493    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:10.812962 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:10.812932    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd2cb00_d129_43c0_b481_07f53d9ad7f3.slice/crio-3dc67ae4b4093f4f266d0fdbdc5f0ccc152b266d760b8cd43b34688f5f40b555 WatchSource:0}: Error finding container 3dc67ae4b4093f4f266d0fdbdc5f0ccc152b266d760b8cd43b34688f5f40b555: Status 404 returned error can't find the container with id 3dc67ae4b4093f4f266d0fdbdc5f0ccc152b266d760b8cd43b34688f5f40b555
Oct 30 03:01:10.815821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:10.815798    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:11.186785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:11.186699    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" event={"ID":"6dd2cb00-d129-43c0-b481-07f53d9ad7f3","Type":"ContainerStarted","Data":"3dc67ae4b4093f4f266d0fdbdc5f0ccc152b266d760b8cd43b34688f5f40b555"}
Oct 30 03:01:12.197826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:12.197794    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" event={"ID":"6dd2cb00-d129-43c0-b481-07f53d9ad7f3","Type":"ContainerStarted","Data":"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"}
Oct 30 03:01:12.198245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:12.197862    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:12.241136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:12.241080    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" podStartSLOduration=1.830751241 podStartE2EDuration="2.241066778s" totalImagesPullingTime="410.315537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:12.238184242 +0000 UTC m=+1211.312355036" watchObservedRunningTime="2026-10-30 03:01:12.241066778 +0000 UTC m=+1211.315237571"
Oct 30 03:01:13.810310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.806390    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:01:13.829738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.829707    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:13.843446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.839782    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:01:13.845264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.845200    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:13.865645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.865605    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:01:13.865917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.865900    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:01:13.905269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:13.889814    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wg9w9\""
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.033859    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.033918    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034009    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034059    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td98g\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034090    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034151    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034181    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034243    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034300    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034325    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034395    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034437    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034490    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034553    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6s97\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034698    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.034730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034738    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.035753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.034791    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.056309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.043182    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:01:14.136548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136444    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.136548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136515    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td98g\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.136548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136547    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136636    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136677    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136730    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136756    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136810    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.136844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136841    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136881    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.136937    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137029    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s97\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137065    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137106    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.137212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137154    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.137586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137244    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.137586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.137287    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.138202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.138171    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.138377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.138354    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.138492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.138371    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.138581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.138430    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.138716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.138696    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.139474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.139439    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.139474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.139450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.139474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.139462    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.139792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.139764    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.140148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.140120    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.145772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.145726    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.145772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.145738    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.145968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.145945    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.148291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.148197    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.161775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.161719    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s97\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.169967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.169934    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.174188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.174160    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zprng\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.175051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.174995    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td98g\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6h2wl\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.316145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.316110    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:14.356767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.356730    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:14.648953 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:14.648915    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a26e7b8_42fe_408d_b5bd_f22332f208c4.slice/crio-26c849d87240794c8a08d759f22c6226d1af80d49ca8abda1116ba916a752e67 WatchSource:0}: Error finding container 26c849d87240794c8a08d759f22c6226d1af80d49ca8abda1116ba916a752e67: Status 404 returned error can't find the container with id 26c849d87240794c8a08d759f22c6226d1af80d49ca8abda1116ba916a752e67
Oct 30 03:01:14.651383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.651355    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:01:14.651608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.651576    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:14.651662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.651639    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:14.651698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.651669    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:14.688155 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:14.688123    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbe6f81_793d_4a2f_be93_856cf3405fbf.slice/crio-d0bdb34fe2af534b68b36077769000a874fff167888c3b104ded99221bc34c5b WatchSource:0}: Error finding container d0bdb34fe2af534b68b36077769000a874fff167888c3b104ded99221bc34c5b: Status 404 returned error can't find the container with id d0bdb34fe2af534b68b36077769000a874fff167888c3b104ded99221bc34c5b
Oct 30 03:01:14.691143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.691117    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:01:14.691506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.691476    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:14.691600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.691553    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:14.691664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:14.691595    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:15.234790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.234734    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" event={"ID":"2a26e7b8-42fe-408d-b5bd-f22332f208c4","Type":"ContainerStarted","Data":"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"}
Oct 30 03:01:15.236331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.236280    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" event={"ID":"2a26e7b8-42fe-408d-b5bd-f22332f208c4","Type":"ContainerStarted","Data":"26c849d87240794c8a08d759f22c6226d1af80d49ca8abda1116ba916a752e67"}
Oct 30 03:01:15.241204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.241171    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" event={"ID":"3dbe6f81-793d-4a2f-be93-856cf3405fbf","Type":"ContainerStarted","Data":"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"}
Oct 30 03:01:15.241414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.241396    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" event={"ID":"3dbe6f81-793d-4a2f-be93-856cf3405fbf","Type":"ContainerStarted","Data":"d0bdb34fe2af534b68b36077769000a874fff167888c3b104ded99221bc34c5b"}
Oct 30 03:01:15.318678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.318645    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:15.320110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.320072    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:15.320329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.320295    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:15.323646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.323602    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podStartSLOduration=2.323589765 podStartE2EDuration="2.323589765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:15.303530459 +0000 UTC m=+1214.377701279" watchObservedRunningTime="2026-10-30 03:01:15.323589765 +0000 UTC m=+1214.397760558"
Oct 30 03:01:15.395254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.394672    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:15.395597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.395554    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podStartSLOduration=2.3955407 podStartE2EDuration="2.3955407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:15.392297416 +0000 UTC m=+1214.466468210" watchObservedRunningTime="2026-10-30 03:01:15.3955407 +0000 UTC m=+1214.469711493"
Oct 30 03:01:15.399655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.397030    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:15.399655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:15.397089    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:16.316621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:16.316590    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:16.317076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:16.317052    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:16.391079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:16.391051    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:16.391356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:16.391318    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:17.317492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:17.317458    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:17.317959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:17.317506    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:17.368251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:17.366068    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:17.368251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:17.366127    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:18.214548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:18.214519    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:18.318294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:18.316817    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:18.318294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:18.316877    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:18.424344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:18.423525    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:18.424344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:18.423597    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:19.316572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:19.316535    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:19.316761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:19.316592    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:19.369251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:19.363886    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:19.369251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:19.363947    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:20.316958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:20.316920    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:20.317427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:20.317388    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:20.365253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:20.362950    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:20.365253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:20.363022    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:21.317488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:21.317441    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:21.318105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:21.317508    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:21.357212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:21.357175    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:21.357489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:21.357461    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:22.316722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.316672    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 03:01:22.316922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.316760    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 03:01:22.358137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.358101    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 03:01:22.358690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.358175    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 03:01:22.383167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.383128    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:01:22.389759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.389731    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.411100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.411077    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"openshift-service-ca.crt\""
Oct 30 03:01:22.411379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.411361    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"kube-root-ca.crt\""
Oct 30 03:01:22.420885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.420864    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"default-dockercfg-28ksf\""
Oct 30 03:01:22.421016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.420968    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"distinct-2-w1-e11f03-kserve-self-signed-certs\""
Oct 30 03:01:22.438290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.438210    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:01:22.535663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.535627    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.535973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.535949    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.536308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.536286    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.536556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.536538    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.536768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.536747    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2ws\" (UniqueName: \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.536932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.536916    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.647248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.647129    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.647512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.647489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2ws\" (UniqueName: \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.647665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.647648    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.647922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.647903    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.648046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.648030    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.648390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.648365    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.649642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.649613    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.650853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.650313    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.651000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.650525    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.688443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.688406    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.722284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.719294    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:22.723219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:22.723188    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2ws\" (UniqueName: \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\") pod \"distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") " pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:23.012350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:23.012219    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:23.400627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:23.400590    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:01:23.406218 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:23.406186    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcc5f5b9_e941_4236_aae5_3c4059863209.slice/crio-10527c414122e6f42c8c951919016b2562f5d7e15f2e3ffb31abbe49b704ed03 WatchSource:0}: Error finding container 10527c414122e6f42c8c951919016b2562f5d7e15f2e3ffb31abbe49b704ed03: Status 404 returned error can't find the container with id 10527c414122e6f42c8c951919016b2562f5d7e15f2e3ffb31abbe49b704ed03
Oct 30 03:01:23.571259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:23.570794    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" event={"ID":"fcc5f5b9-e941-4236-aae5-3c4059863209","Type":"ContainerStarted","Data":"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"}
Oct 30 03:01:23.571259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:23.570851    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" event={"ID":"fcc5f5b9-e941-4236-aae5-3c4059863209","Type":"ContainerStarted","Data":"10527c414122e6f42c8c951919016b2562f5d7e15f2e3ffb31abbe49b704ed03"}
Oct 30 03:01:23.616539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:23.616470    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podStartSLOduration=1.6164488179999998 podStartE2EDuration="1.616448818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:23.614341738 +0000 UTC m=+1222.688512538" watchObservedRunningTime="2026-10-30 03:01:23.616448818 +0000 UTC m=+1222.690619613"
Oct 30 03:01:24.012659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.012620    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:24.419249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.414424    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:01:24.419249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.414572    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:01:24.444619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.444586    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zprng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:01:24.444874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.444845    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:01:24.616315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.616282    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:24.616873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.616845    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:24.618436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.618411    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:01:24.632551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.632527    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:24.633089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.633062    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:24.634367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:24.634347    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:01:28.826668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.826635    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:01:28.832393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.832368    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.922573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.922541    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:01:28.942295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.934189    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.942644    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.942811    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.942842    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.942876    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.942949    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69zc\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.943041    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:28.952080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:28.943096    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.011717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.011433    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116083    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116173    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116207    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116348    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116372    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116405    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116469    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69zc\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116543    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.116996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.116590    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.117656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.117241    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.117656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.117512    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.118421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.118144    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.119066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.119043    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.119320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.119294    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.182029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.181987    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.189401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.187264    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69zc\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.189401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.188476    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.191138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.191110    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kpx2s\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:29.220633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:29.220601    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:30.259825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.259792    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:01:30.365735 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:30.348534    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fa08d2_b0c5_4e69_84b9_590f404b6540.slice/crio-bf9b0a9925271b0887a90db7dbddf720ac2d1ae0baa60ad9d081513a80012a22 WatchSource:0}: Error finding container bf9b0a9925271b0887a90db7dbddf720ac2d1ae0baa60ad9d081513a80012a22: Status 404 returned error can't find the container with id bf9b0a9925271b0887a90db7dbddf720ac2d1ae0baa60ad9d081513a80012a22
Oct 30 03:01:30.393881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.393752    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:30.394109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.394041    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:30.394390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.394337    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:30.468281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.463350    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:01:30.671988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.671947    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:01:30.758256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.731013    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.789255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.786144    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827291    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827367    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827415    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827533    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827571    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827716    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltph7\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827770    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.827889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.848251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.828056    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:30.927319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.887008    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:01:31.043349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.954386    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.055982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:30.961847    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.056757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.056728    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.158145    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.158341    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.158625    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.158804    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltph7\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.159050    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.159201    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.199724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.159336    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.259766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.105151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.318254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.149996    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.386287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.334852    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.386287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.348132    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.386287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.348514    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.386287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.349541    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.405000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.402721    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.411522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.411487    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.411953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.411928    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.412126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.412112    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.412271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.412257    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.412462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.412439    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.412687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.412632    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.413014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.412826    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xwj\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.413211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.413192    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.413477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.413459    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.415534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.415511    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.424900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.424854    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltph7\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.428843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.428819    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:31.517634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.517592    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.517952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.517932    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.520355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.520319    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.520674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.520650    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.520864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.520849    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.521072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.521044    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.521659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.519977    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.522072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.522051    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.523531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.522191    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xwj\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.523729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.523709    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.523994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.523961    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.524480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.523397    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.525501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.525479    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.526796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.526744    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.531849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.531495    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:01:31.533145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.533104    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" event={"ID":"26fa08d2-b0c5-4e69-84b9-590f404b6540","Type":"ContainerStarted","Data":"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"}
Oct 30 03:01:31.533331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.533310    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" event={"ID":"26fa08d2-b0c5-4e69-84b9-590f404b6540","Type":"ContainerStarted","Data":"bf9b0a9925271b0887a90db7dbddf720ac2d1ae0baa60ad9d081513a80012a22"}
Oct 30 03:01:31.534370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.534347    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.550258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.541223    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.553669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.553619    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.560463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.560405    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xwj\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.610337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.610299    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:31.619274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.618544    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podStartSLOduration=3.618529867 podStartE2EDuration="3.618529867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:31.616966494 +0000 UTC m=+1230.691137288" watchObservedRunningTime="2026-10-30 03:01:31.618529867 +0000 UTC m=+1230.692700660"
Oct 30 03:01:31.989865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.989825    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:31.990000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.989916    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:31.990000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:31.989965    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:32.001055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.001019    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:01:32.076938 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:32.076890    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff07ceb_8b79_4ccf_8b28_acce23859fcf.slice/crio-726be06597c913440133c4c5e23c941aca4e6968144a493f39c471196cd86a3a WatchSource:0}: Error finding container 726be06597c913440133c4c5e23c941aca4e6968144a493f39c471196cd86a3a: Status 404 returned error can't find the container with id 726be06597c913440133c4c5e23c941aca4e6968144a493f39c471196cd86a3a
Oct 30 03:01:32.080653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.080632    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:01:32.081992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.081618    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:32.081992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.081678    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:32.081992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.081705    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:32.221968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.221889    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:32.222943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.222924    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:32.223097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.223076    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:32.593455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.593415    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" event={"ID":"7ff07ceb-8b79-4ccf-8b28-acce23859fcf","Type":"ContainerStarted","Data":"c15024eaa7a394354e0969593a31cee0c757ab49804b2e59b3106a6733b4184c"}
Oct 30 03:01:32.604566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.600338    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" event={"ID":"7ff07ceb-8b79-4ccf-8b28-acce23859fcf","Type":"ContainerStarted","Data":"726be06597c913440133c4c5e23c941aca4e6968144a493f39c471196cd86a3a"}
Oct 30 03:01:32.626250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.623703    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" event={"ID":"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd","Type":"ContainerStarted","Data":"d7db3b1e5220b5d1d5570e4ec9e24ce65cf92eed50f6b41a1012ce231fc35f96"}
Oct 30 03:01:32.626250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.623766    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" event={"ID":"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd","Type":"ContainerStarted","Data":"3047b996af70157651e62692f56ebe5c0b1b00e230e06dd478b1aad74aecb5de"}
Oct 30 03:01:32.626250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.623821    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:32.699329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.690716    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podStartSLOduration=2.690691921 podStartE2EDuration="2.690691921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:32.668846666 +0000 UTC m=+1231.743017467" watchObservedRunningTime="2026-10-30 03:01:32.690691921 +0000 UTC m=+1231.764862717"
Oct 30 03:01:32.733106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:32.733053    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podStartSLOduration=2.73303838 podStartE2EDuration="2.73303838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:32.726732645 +0000 UTC m=+1231.800903438" watchObservedRunningTime="2026-10-30 03:01:32.73303838 +0000 UTC m=+1231.807209172"
Oct 30 03:01:33.015541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.014142    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:33.126841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.126812    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:33.231631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.230626    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.231631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.230690    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:33.448252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.442124    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:33.448252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.442262    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.448252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.442316    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:33.731794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.731525    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.732553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:33.732518    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:34.227256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.226642    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.227256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.226724    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:34.431712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.431674    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.432010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.431979    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:34.577662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.577631    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:01:34.621718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.621181    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.621718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:34.621263    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:35.224252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.223920    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.224252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.223986    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:35.435648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.434568    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.435648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.434628    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:35.614577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.614059    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.614577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:35.614159    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:36.221717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.221682    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.222028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.221982    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:36.430033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.429604    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.430033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.429664    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:36.611421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.610962    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.611421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:36.611028    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:37.226115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.225540    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.226115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.225599    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:37.431355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.430548    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.431355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.430609    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:37.628296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.626663    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.628296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:37.626727    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:38.226257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.225976    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:38.226257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.226042    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:38.411327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.393298    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"]
Oct 30 03:01:38.532213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.464789    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:38.532213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.464851    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:38.601256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.536687    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.727284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.715183    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"]
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.758550    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.729157    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.763461    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.764593    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkd8\" (UniqueName: \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.764783    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.764942    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.785808    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:01:38.818320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.785880    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:01:38.881363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.876509    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkd8\" (UniqueName: \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.881363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.876679    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.919359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.918513    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:38.919359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.918634    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:01:38.919621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.919362    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:01:38.919902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.919854    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:39.007538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:38.998809    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:39.034197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.033969    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkd8\" (UniqueName: \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\") pod \"maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:39.225885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.225366    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:39.225885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.225452    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:39.394254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.391362    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:39.403501 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:01:39.403436    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:01:39.431339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.431092    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:39.431339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.431147    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:39.624987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.624585    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:39.624987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:39.624637    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:40.001352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.001147    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"]
Oct 30 03:01:40.242305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.229469    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:40.242305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.229529    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:40.543173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.511758    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:01:40.543173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.511837    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:01:40.583598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.583560    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb" event={"ID":"4c64d87c-9213-4e7b-802a-0ed57008acb2","Type":"ContainerStarted","Data":"04ad03a8d9316b72ec1a4b26916f35588f5d1b8ff677aec2ccd871c5793bc232"}
Oct 30 03:01:40.617573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.617536    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:01:40.617773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:40.617597    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 03:01:41.227248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.226748    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 03:01:41.227248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.226810    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 03:01:41.663249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.647082    4214 generic.go:356] "Generic (PLEG): container finished" podID="4c64d87c-9213-4e7b-802a-0ed57008acb2" containerID="0fe12010f6dc6d6597c28cb8562ab28865232f2d27c99f710020e64bceaf7bb4" exitCode=0
Oct 30 03:01:41.663249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.647328    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb" event={"ID":"4c64d87c-9213-4e7b-802a-0ed57008acb2","Type":"ContainerDied","Data":"0fe12010f6dc6d6597c28cb8562ab28865232f2d27c99f710020e64bceaf7bb4"}
Oct 30 03:01:41.899102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.897847    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:41.899102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.898894    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:41.899945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:41.899917    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:01:42.054371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.052164    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:42.058294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.056198    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:42.065756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.065610    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:01:42.951023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.950626    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:42.995722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.992571    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:42.995722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.993914    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:42.997386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:42.996135    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:01:43.084381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.084340    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\") pod \"4c64d87c-9213-4e7b-802a-0ed57008acb2\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") "
Oct 30 03:01:43.084690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.084447    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\") pod \"4c64d87c-9213-4e7b-802a-0ed57008acb2\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") "
Oct 30 03:01:43.084690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.084661    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\" (UniqueName: \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\") pod \"4c64d87c-9213-4e7b-802a-0ed57008acb2\" (UID: \"4c64d87c-9213-4e7b-802a-0ed57008acb2\") "
Oct 30 03:01:43.086644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.086594    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca" pod "4c64d87c-9213-4e7b-802a-0ed57008acb2" (UID: "4c64d87c-9213-4e7b-802a-0ed57008acb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:43.089350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.089309    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8" pod "4c64d87c-9213-4e7b-802a-0ed57008acb2" (UID: "4c64d87c-9213-4e7b-802a-0ed57008acb2"). InnerVolumeSpecName "kube-api-access-glkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:43.090406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.090377    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca" pod "4c64d87c-9213-4e7b-802a-0ed57008acb2" (UID: "4c64d87c-9213-4e7b-802a-0ed57008acb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:01:43.187198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.187119    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkd8\" (UniqueName: \"kubernetes.io/projected/4c64d87c-9213-4e7b-802a-0ed57008acb2-kube-api-access-glkd8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:43.187198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.187159    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:43.187198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.187173    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c64d87c-9213-4e7b-802a-0ed57008acb2-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:43.685474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.685443    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"
Oct 30 03:01:43.685698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.685493    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb" event={"ID":"4c64d87c-9213-4e7b-802a-0ed57008acb2","Type":"ContainerDied","Data":"04ad03a8d9316b72ec1a4b26916f35588f5d1b8ff677aec2ccd871c5793bc232"}
Oct 30 03:01:43.685698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:43.685538    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04ad03a8d9316b72ec1a4b26916f35588f5d1b8ff677aec2ccd871c5793bc232"
Oct 30 03:01:45.706468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.706435    4214 generic.go:356] "Generic (PLEG): container finished" podID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerID="48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7" exitCode=0
Oct 30 03:01:45.706901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.706505    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerDied","Data":"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"}
Oct 30 03:01:45.706958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.706944    4214 scope.go:122] "RemoveContainer" containerID="48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"
Oct 30 03:01:45.708669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.708612    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:45.708771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.708710    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:45.708771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:45.708744    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:46.734535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:46.734492    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerStarted","Data":"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"}
Oct 30 03:01:47.570797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:47.570759    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:01:47.571691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:47.571669    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:47.571759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:47.571712    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:48.572758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:48.572719    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:48.573312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:48.572774    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:49.583268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:49.574315    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:49.583268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:49.574388    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:50.570974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:50.570929    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:50.571177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:50.570996    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:51.578278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:51.578223    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:51.579069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:51.579035    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:52.609419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:52.602482    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:01:52.675870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:52.675824    4214 patch_prober.go:29] interesting pod/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 03:01:52.676580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:52.676547    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 03:01:53.086902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.086868    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:53.088487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.088439    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerName="maas-api" containerID="cri-o://31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d" gracePeriod=30
Oct 30 03:01:53.209383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.209335    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.139:8443/health\": dial tcp 10.133.0.139:8443: connect: connection refused"
Oct 30 03:01:53.748881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.748789    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:53.812344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.812310    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\") pod \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") "
Oct 30 03:01:53.812747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.812725    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\") pod \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") "
Oct 30 03:01:53.813075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.813057    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\" (UniqueName: \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\") pod \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") "
Oct 30 03:01:53.819391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.819355    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\") pod \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\" (UID: \"6dd2cb00-d129-43c0-b481-07f53d9ad7f3\") "
Oct 30 03:01:53.845307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.824459    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca" pod "6dd2cb00-d129-43c0-b481-07f53d9ad7f3" (UID: "6dd2cb00-d129-43c0-b481-07f53d9ad7f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:53.845307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.835950    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls" pod "6dd2cb00-d129-43c0-b481-07f53d9ad7f3" (UID: "6dd2cb00-d129-43c0-b481-07f53d9ad7f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:01:53.848510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.848473    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:01:53.870579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.854826    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs" pod "6dd2cb00-d129-43c0-b481-07f53d9ad7f3" (UID: "6dd2cb00-d129-43c0-b481-07f53d9ad7f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:01:53.915122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.913596    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:01:53.915122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.914057    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb" pod "6dd2cb00-d129-43c0-b481-07f53d9ad7f3" (UID: "6dd2cb00-d129-43c0-b481-07f53d9ad7f3"). InnerVolumeSpecName "kube-api-access-rbnfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:53.947332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.946928    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbnfb\" (UniqueName: \"kubernetes.io/projected/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-kube-api-access-rbnfb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:53.947332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.946976    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:53.947332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.946993    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:53.947332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.947008    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd2cb00-d129-43c0-b481-07f53d9ad7f3-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:01:53.958279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.955708    4214 generic.go:356] "Generic (PLEG): container finished" podID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerID="31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d" exitCode=0
Oct 30 03:01:53.958279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.957003    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"
Oct 30 03:01:53.959353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.958825    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" event={"ID":"6dd2cb00-d129-43c0-b481-07f53d9ad7f3","Type":"ContainerDied","Data":"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"}
Oct 30 03:01:53.959353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.958874    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb" event={"ID":"6dd2cb00-d129-43c0-b481-07f53d9ad7f3","Type":"ContainerDied","Data":"3dc67ae4b4093f4f266d0fdbdc5f0ccc152b266d760b8cd43b34688f5f40b555"}
Oct 30 03:01:53.959353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:53.958911    4214 scope.go:122] "RemoveContainer" containerID="31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"
Oct 30 03:01:54.029276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.024496    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"]
Oct 30 03:01:54.029276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.026881    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ca6fbc2-cgbpb"]
Oct 30 03:01:54.085258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.079322    4214 scope.go:122] "RemoveContainer" containerID="31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"
Oct 30 03:01:54.091251 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:01:54.090833    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d\": container with ID starting with 31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d not found: ID does not exist" containerID="31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"
Oct 30 03:01:54.091251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.090882    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d"} err="failed to get container status \"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d\": rpc error: code = NotFound desc = could not find container \"31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d\": container with ID starting with 31bff0749511c1ddf20ab09f9b5657e4627914c2fa8f2b166ad767c6eba8991d not found: ID does not exist"
Oct 30 03:01:54.253980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.253909    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:54.476394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.464036    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ca6fbc2-9d5fc755-wlzfb"]
Oct 30 03:01:54.494262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.484070    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:01:54.971106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:54.971071    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy" containerID="cri-o://03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d" gracePeriod=30
Oct 30 03:01:55.069282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.069247    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c64d87c-9213-4e7b-802a-0ed57008acb2" path="/var/lib/kubelet/pods/4c64d87c-9213-4e7b-802a-0ed57008acb2/volumes"
Oct 30 03:01:55.069750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.069733    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" path="/var/lib/kubelet/pods/6dd2cb00-d129-43c0-b481-07f53d9ad7f3/volumes"
Oct 30 03:01:55.415751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.415712    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:01:55.416936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.416915    4214 cpu_manager.go:414] "containerMap: removing container" podUID="4c64d87c-9213-4e7b-802a-0ed57008acb2" containerName="revoke-keys"
Oct 30 03:01:55.417007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.416939    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c64d87c-9213-4e7b-802a-0ed57008acb2" containerName="revoke-keys"
Oct 30 03:01:55.417007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.416973    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerName="maas-api"
Oct 30 03:01:55.417007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.416981    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerName="maas-api"
Oct 30 03:01:55.417155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.417144    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c64d87c-9213-4e7b-802a-0ed57008acb2" containerName="revoke-keys"
Oct 30 03:01:55.417191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.417162    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd2cb00-d129-43c0-b481-07f53d9ad7f3" containerName="maas-api"
Oct 30 03:01:55.421939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.421912    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.447204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.447172    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6b5f19-openshift-default-dockercfg-ks468\""
Oct 30 03:01:55.448963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.448936    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:01:55.451647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451601    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.451779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451697    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.451779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451770    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451822    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skjm6\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451926    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.451982    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.452021    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.452415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.452179    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.570257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.559376    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.581449    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skjm6\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.581978    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582048    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582136    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582176    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582372    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582569    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.583251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.582706    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.591313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.590606    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.591313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.590913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.591313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.591157    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.627253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.593435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.627253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.624132    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.638254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.629976    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.638254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.630506    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.638254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.636555    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.661277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.652999    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skjm6\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:55.761299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:55.756577    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:56.504708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:56.504083    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:01:56.506414 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:01:56.506377    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ad089a_84e3_4867_81c4_34890dc9e56e.slice/crio-14d03b7a519b70caf693bde5907956e055600134b296dbb4e7b896f613a20396 WatchSource:0}: Error finding container 14d03b7a519b70caf693bde5907956e055600134b296dbb4e7b896f613a20396: Status 404 returned error can't find the container with id 14d03b7a519b70caf693bde5907956e055600134b296dbb4e7b896f613a20396
Oct 30 03:01:56.509459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:56.509423    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:56.509553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:56.509493    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:56.509553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:56.509525    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:01:57.046769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:57.046731    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerStarted","Data":"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"}
Oct 30 03:01:57.046969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:57.046797    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerStarted","Data":"14d03b7a519b70caf693bde5907956e055600134b296dbb4e7b896f613a20396"}
Oct 30 03:01:57.116945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:57.116890    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podStartSLOduration=2.116876302 podStartE2EDuration="2.116876302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:57.109853672 +0000 UTC m=+1256.184024466" watchObservedRunningTime="2026-10-30 03:01:57.116876302 +0000 UTC m=+1256.191047093"
Oct 30 03:01:57.757535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:57.757489    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:57.763542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:57.763514    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:58.060604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:58.060574    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:01:58.061790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:01:58.061764    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:02:00.406809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.406778    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:02:00.509209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509172    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509247    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509291    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509343    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509402    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509491    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509670    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.509837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.509816    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.510123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.510085    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:02:00.510224 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.510083    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\") pod \"8b391872-f100-4ba6-8ae9-23f33d434d5d\" (UID: \"8b391872-f100-4ba6-8ae9-23f33d434d5d\") "
Oct 30 03:02:00.511134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.511110    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.511913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.511877    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:02:00.512137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.512106    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:02:00.512808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.512774    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "kube-api-access-f5zt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:02:00.514169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.514078    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:02:00.514169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.514131    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:02:00.514169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.514157    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:02:00.514359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.514285    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:02:00.515548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.515527    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token" pod "8b391872-f100-4ba6-8ae9-23f33d434d5d" (UID: "8b391872-f100-4ba6-8ae9-23f33d434d5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612600    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zt5\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-kube-api-access-f5zt5\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612631    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612642    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612650    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612658    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b391872-f100-4ba6-8ae9-23f33d434d5d-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612665    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612675    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b391872-f100-4ba6-8ae9-23f33d434d5d-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:00.612684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:00.612683    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b391872-f100-4ba6-8ae9-23f33d434d5d-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:01.096764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.096732    4214 generic.go:356] "Generic (PLEG): container finished" podID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerID="03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d" exitCode=0
Oct 30 03:02:01.096905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.096779    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"
Oct 30 03:02:01.096905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.096808    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerDied","Data":"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"}
Oct 30 03:02:01.096905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.096858    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b" event={"ID":"8b391872-f100-4ba6-8ae9-23f33d434d5d","Type":"ContainerDied","Data":"057c88610dbff8752892b9d558b09955c4232c6c252b897fa63525f54c0b19b5"}
Oct 30 03:02:01.096905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.096879    4214 scope.go:122] "RemoveContainer" containerID="03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"
Oct 30 03:02:01.109147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.109125    4214 scope.go:122] "RemoveContainer" containerID="48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"
Oct 30 03:02:01.122290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.122251    4214 scope.go:122] "RemoveContainer" containerID="03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"
Oct 30 03:02:01.122596 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:02:01.122572    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d\": container with ID starting with 03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d not found: ID does not exist" containerID="03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"
Oct 30 03:02:01.122684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.122603    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d"} err="failed to get container status \"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d\": rpc error: code = NotFound desc = could not find container \"03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d\": container with ID starting with 03918682a3ff8b72d8d8e97850d29de2dd685192d7f2af6b9b04e203c993330d not found: ID does not exist"
Oct 30 03:02:01.122684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.122622    4214 scope.go:122] "RemoveContainer" containerID="48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"
Oct 30 03:02:01.122921 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:02:01.122901    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7\": container with ID starting with 48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7 not found: ID does not exist" containerID="48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"
Oct 30 03:02:01.123010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.122926    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7"} err="failed to get container status \"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7\": rpc error: code = NotFound desc = could not find container \"48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7\": container with ID starting with 48a4f97eb8f1a0814d9e237963313c0f5ed61670b33eb6bbc6ad9fe0ceef7fe7 not found: ID does not exist"
Oct 30 03:02:01.171734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.171645    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:02:01.184919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.184859    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9ca6fbc2-gw-openshift-default-6cf8844885-4lr9b"]
Oct 30 03:02:01.613199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.613165    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"]
Oct 30 03:02:01.614203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.614172    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.614203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.614204    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.614490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.614462    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.614609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.614592    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.637251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.634686    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.648768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.648738    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"]
Oct 30 03:02:01.650531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.650506    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:02:01.650647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.650540    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:02:01.723715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.723660    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:02:01.725021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.724993    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.725021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.725022    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" containerName="istio-proxy"
Oct 30 03:02:01.728794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.728773    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:01.743307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.741437    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qgp\" (UniqueName: \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.743307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.741522    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.747368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.747118    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.748012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.747988    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-6b5f19-dockercfg-298pw\""
Oct 30 03:02:01.748702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.748325    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-6b5f19\""
Oct 30 03:02:01.761488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.758383    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:02:01.774503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.774465    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:02:01.779190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.779167    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:01.813454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.813421    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:02:01.856626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.855099    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:01.856626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.855205    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.856626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.855247    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpq9\" (UniqueName: \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:01.856626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.855518    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qgp\" (UniqueName: \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.856626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.855583    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.893766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.891244    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:01.904891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:01.895958    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.154783    4214 scope.go:122] "RemoveContainer" containerID="1bb322577e709071a174f3b1dd22ab7a20bf14d7433b22b7a54be8b3f71fb891"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.184722    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.184817    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpq9\" (UniqueName: \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.184990    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.185086    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlx95\" (UniqueName: \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.193303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.187104    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:02.210253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.206798    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qgp\" (UniqueName: \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\") pod \"maas-api-revoke-sub-1a273f80-gh5v6\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:02.210253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.208932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpq9\" (UniqueName: \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\") pod \"payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:02.267722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.261372    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:02.311279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.302489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.330257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.303658    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.330257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.327316    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlx95\" (UniqueName: \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.366405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.366374    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:02.462327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.428212    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlx95\" (UniqueName: \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\") pod \"payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.462327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.451437    4214 scope.go:122] "RemoveContainer" containerID="f347e7af25fd1db158bd1c627b53afb3e81f0319007020381c69ab482b99f79b"
Oct 30 03:02:02.545549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.545429    4214 scope.go:122] "RemoveContainer" containerID="56074c74ba1fa72fcff3f6398972aa3c72460c93b47d826304331532a6fd942f"
Oct 30 03:02:02.562928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.562881    4214 scope.go:122] "RemoveContainer" containerID="69db92b916efdee8bfe11a3964319010ceabf86f173808908389c4e247c803f1"
Oct 30 03:02:02.575823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.575710    4214 scope.go:122] "RemoveContainer" containerID="b0941805999f4357993fb48f1746a9ee2222683237310c3b5e8876340ef3018b"
Oct 30 03:02:02.595978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.595558    4214 scope.go:122] "RemoveContainer" containerID="b0c1cba1ba1332bff4b5174321c2c4e77a69a51574af4dcf7f5127f6936c2fd7"
Oct 30 03:02:02.615791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.615763    4214 scope.go:122] "RemoveContainer" containerID="6ea38a35ed2481dc2a1f41f6b533a205f92195d5ec8d60b88a69f67dc4614ef6"
Oct 30 03:02:02.706387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.700610    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:02.860132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.860103    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"]
Oct 30 03:02:02.877296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:02.876753    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:02:03.032544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.032514    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"]
Oct 30 03:02:03.040374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.040342    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5326539f-6v6df"]
Oct 30 03:02:03.040532 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:02:03.040508    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e89013a_7eba_4c7a_bb9e_46d8dd72cb4e.slice/crio-60d002fca1a056dff6813c4c4ffa41407730ac2c4f3c1dfaaa1834f4b2e62fed WatchSource:0}: Error finding container 60d002fca1a056dff6813c4c4ffa41407730ac2c4f3c1dfaaa1834f4b2e62fed: Status 404 returned error can't find the container with id 60d002fca1a056dff6813c4c4ffa41407730ac2c4f3c1dfaaa1834f4b2e62fed
Oct 30 03:02:03.046201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.046178    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:02:03.071331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.071298    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acdf987-90a7-4d9e-98f6-9895ff22ddd2" path="/var/lib/kubelet/pods/1acdf987-90a7-4d9e-98f6-9895ff22ddd2/volumes"
Oct 30 03:02:03.071865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.071848    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b391872-f100-4ba6-8ae9-23f33d434d5d" path="/var/lib/kubelet/pods/8b391872-f100-4ba6-8ae9-23f33d434d5d/volumes"
Oct 30 03:02:03.216760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.216678    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" event={"ID":"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e","Type":"ContainerStarted","Data":"60d002fca1a056dff6813c4c4ffa41407730ac2c4f3c1dfaaa1834f4b2e62fed"}
Oct 30 03:02:03.219510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.219482    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" event={"ID":"e0620af5-4d12-41b6-8528-5b51cea0f3f0","Type":"ContainerStarted","Data":"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"}
Oct 30 03:02:03.219681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.219536    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" event={"ID":"e0620af5-4d12-41b6-8528-5b51cea0f3f0","Type":"ContainerStarted","Data":"6abd7c6faed3e6f77ced5e2b76d40bc4e5dcf098427171373405f2e794507433"}
Oct 30 03:02:03.219816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.219771    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:03.221498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.221473    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6" event={"ID":"121b90cb-544e-4a6d-ba94-74937c0c865a","Type":"ContainerStarted","Data":"d6337e7fe78a2a1163d129cf8c679d00bbcde99c7e71e7e0e2e3c443ee8e6530"}
Oct 30 03:02:03.266008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:03.265955    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" podStartSLOduration=2.265942423 podStartE2EDuration="2.265942423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:03.265638425 +0000 UTC m=+1262.339809218" watchObservedRunningTime="2026-10-30 03:02:03.265942423 +0000 UTC m=+1262.340113215"
Oct 30 03:02:04.290390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:04.286700    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" event={"ID":"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e","Type":"ContainerStarted","Data":"418db4de87babcdf886cac8cc71248faf2bdbf3ed9def61127b5a26754ba3ba3"}
Oct 30 03:02:04.303254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:04.293958    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:04.328348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:04.328318    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6" event={"ID":"121b90cb-544e-4a6d-ba94-74937c0c865a","Type":"ContainerDied","Data":"eebc693c24efd2402929f36a8aa3d65adea567bb4105edc48a88b3b11727c99c"}
Oct 30 03:02:04.328556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:04.328028    4214 generic.go:356] "Generic (PLEG): container finished" podID="121b90cb-544e-4a6d-ba94-74937c0c865a" containerID="eebc693c24efd2402929f36a8aa3d65adea567bb4105edc48a88b3b11727c99c" exitCode=0
Oct 30 03:02:04.416066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:04.415952    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" podStartSLOduration=3.415922837 podStartE2EDuration="3.415922837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:04.379598246 +0000 UTC m=+1263.453769039" watchObservedRunningTime="2026-10-30 03:02:04.415922837 +0000 UTC m=+1263.490093632"
Oct 30 03:02:05.571031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.570994    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:02:05.577063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.577026    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:05.592694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.592657    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:02:05.599896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.599823    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ac7936-dockercfg-5hq26\""
Oct 30 03:02:05.628134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.628101    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:05.628383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.628363    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ac7936\""
Oct 30 03:02:05.676983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.676938    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\") pod \"121b90cb-544e-4a6d-ba94-74937c0c865a\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") "
Oct 30 03:02:05.677174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.677072    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\" (UniqueName: \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\") pod \"121b90cb-544e-4a6d-ba94-74937c0c865a\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") "
Oct 30 03:02:05.677705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.677665    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\") pod \"121b90cb-544e-4a6d-ba94-74937c0c865a\" (UID: \"121b90cb-544e-4a6d-ba94-74937c0c865a\") "
Oct 30 03:02:05.678162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.678136    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca" pod "121b90cb-544e-4a6d-ba94-74937c0c865a" (UID: "121b90cb-544e-4a6d-ba94-74937c0c865a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:02:05.678347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.678182    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:05.678540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.678521    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtm4\" (UniqueName: \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:05.679078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.679054    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:05.704466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.704437    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca" pod "121b90cb-544e-4a6d-ba94-74937c0c865a" (UID: "121b90cb-544e-4a6d-ba94-74937c0c865a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:02:05.718488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.718391    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp" pod "121b90cb-544e-4a6d-ba94-74937c0c865a" (UID: "121b90cb-544e-4a6d-ba94-74937c0c865a"). InnerVolumeSpecName "kube-api-access-s7qgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:02:05.786251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.782367    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtm4\" (UniqueName: \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:05.786251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.782791    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:05.786251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.782893    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b90cb-544e-4a6d-ba94-74937c0c865a-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:05.786251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.782910    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7qgp\" (UniqueName: \"kubernetes.io/projected/121b90cb-544e-4a6d-ba94-74937c0c865a-kube-api-access-s7qgp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:02:05.786251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:05.783832    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:06.081928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:06.080005    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtm4\" (UniqueName: \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\") pod \"payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:06.233258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:06.231194    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:06.517254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:06.509338    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"
Oct 30 03:02:06.517254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:06.515303    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6" event={"ID":"121b90cb-544e-4a6d-ba94-74937c0c865a","Type":"ContainerDied","Data":"d6337e7fe78a2a1163d129cf8c679d00bbcde99c7e71e7e0e2e3c443ee8e6530"}
Oct 30 03:02:06.517254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:06.515402    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6337e7fe78a2a1163d129cf8c679d00bbcde99c7e71e7e0e2e3c443ee8e6530"
Oct 30 03:02:07.130714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:07.129337    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:02:07.592653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:07.587224    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" event={"ID":"630f8512-3070-42d8-8bb3-fea98031b56c","Type":"ContainerStarted","Data":"82a97d22a0ededc1046251673016455ca32eec91f8ef208849b96efb8f027564"}
Oct 30 03:02:07.592653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:07.587328    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" event={"ID":"630f8512-3070-42d8-8bb3-fea98031b56c","Type":"ContainerStarted","Data":"b0ad4e1727f5e88cbbcc69549db3de2b051e91df91653d0e739923da2b2f71fb"}
Oct 30 03:02:07.593614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:07.593589    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:07.676207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:07.676131    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" podStartSLOduration=2.676110592 podStartE2EDuration="2.676110592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:07.652692514 +0000 UTC m=+1266.726863308" watchObservedRunningTime="2026-10-30 03:02:07.676110592 +0000 UTC m=+1266.750281385"
Oct 30 03:02:09.173248 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:02:09.173192    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:02:09.435149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.435068    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:02:09.436300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.436278    4214 cpu_manager.go:414] "containerMap: removing container" podUID="121b90cb-544e-4a6d-ba94-74937c0c865a" containerName="revoke-keys"
Oct 30 03:02:09.436350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.436303    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121b90cb-544e-4a6d-ba94-74937c0c865a" containerName="revoke-keys"
Oct 30 03:02:09.436481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.436469    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="121b90cb-544e-4a6d-ba94-74937c0c865a" containerName="revoke-keys"
Oct 30 03:02:09.468100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.468063    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.482394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.482361    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:02:09.485610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.485581    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-360818-serving-cert\""
Oct 30 03:02:09.570544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.570512    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.570754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.570698    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.570832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.570799    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6p7\" (UniqueName: \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.570902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.570888    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.697242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.672565    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.697242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.672774    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.697242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.672858    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6p7\" (UniqueName: \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.697242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.688650    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.697242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.673018    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.734287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.730069    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.734784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.734750    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.785117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.785035    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6p7\" (UniqueName: \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\") pod \"maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:09.822453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:09.822141    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:10.183154 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:02:10.183121    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7622710_e54b_44c0_befb_12a7f29bedde.slice/crio-efeafb7653a01475427145160080a97fe51a9906670ada75b240de8c3fa86a9a WatchSource:0}: Error finding container efeafb7653a01475427145160080a97fe51a9906670ada75b240de8c3fa86a9a: Status 404 returned error can't find the container with id efeafb7653a01475427145160080a97fe51a9906670ada75b240de8c3fa86a9a
Oct 30 03:02:10.184948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:10.184919    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:02:10.770250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:10.766124    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" event={"ID":"c7622710-e54b-44c0-befb-12a7f29bedde","Type":"ContainerStarted","Data":"efeafb7653a01475427145160080a97fe51a9906670ada75b240de8c3fa86a9a"}
Oct 30 03:02:11.780003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:11.779954    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" event={"ID":"c7622710-e54b-44c0-befb-12a7f29bedde","Type":"ContainerStarted","Data":"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"}
Oct 30 03:02:11.780524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:11.780191    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:11.829842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:11.829790    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" podStartSLOduration=2.277589594 podStartE2EDuration="2.82977753s" totalImagesPullingTime="552.187936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:11.824809542 +0000 UTC m=+1270.898980345" watchObservedRunningTime="2026-10-30 03:02:11.82977753 +0000 UTC m=+1270.903948322"
Oct 30 03:02:14.416576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:14.416548    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:02:16.536938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:16.535433    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:02:17.030942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:17.030904    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"]
Oct 30 03:02:17.046657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:17.046622    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cc9662-fsr4w"]
Oct 30 03:02:17.069878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:17.069841    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d3cc44-5c64-40fe-9028-05e864b938f3" path="/var/lib/kubelet/pods/24d3cc44-5c64-40fe-9028-05e864b938f3/volumes"
Oct 30 03:02:17.797689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:17.797655    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:02:19.531500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.531467    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:02:19.536165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.536143    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.559072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.559046    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:02:19.585251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.561819    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-6b5f19-serving-cert\""
Oct 30 03:02:19.591478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.591442    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.591869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.591841    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxmn\" (UniqueName: \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.592340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.592318    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.592455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.592442    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.685309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.685278    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:02:19.693808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.693776    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.693947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.693838    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxmn\" (UniqueName: \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.693947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.693936    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.694069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.694054    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.694581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.694555    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.696557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.696530    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.696657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.696563    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.776283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.765845    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxmn\" (UniqueName: \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\") pod \"maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:19.874077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:19.874030    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:20.291821 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:02:20.291788    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328703a5_2224_4370_a53c_fb9b59915ab2.slice/crio-9952da68f586cc3f2b066fa698da6560d207201a4ab8d6133545908db2383fa7 WatchSource:0}: Error finding container 9952da68f586cc3f2b066fa698da6560d207201a4ab8d6133545908db2383fa7: Status 404 returned error can't find the container with id 9952da68f586cc3f2b066fa698da6560d207201a4ab8d6133545908db2383fa7
Oct 30 03:02:20.296832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:20.296803    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:02:21.014048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:21.013564    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" event={"ID":"328703a5-2224-4370-a53c-fb9b59915ab2","Type":"ContainerStarted","Data":"9952da68f586cc3f2b066fa698da6560d207201a4ab8d6133545908db2383fa7"}
Oct 30 03:02:22.024406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:22.024361    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" event={"ID":"328703a5-2224-4370-a53c-fb9b59915ab2","Type":"ContainerStarted","Data":"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"}
Oct 30 03:02:22.024879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:22.024775    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:22.066700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:22.066648    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" podStartSLOduration=2.5041525 podStartE2EDuration="3.066631314s" totalImagesPullingTime="562.478814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.06453707 +0000 UTC m=+1281.138707862" watchObservedRunningTime="2026-10-30 03:02:22.066631314 +0000 UTC m=+1281.140802109"
Oct 30 03:02:26.030450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:26.030414    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"]
Oct 30 03:02:26.042657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:26.042631    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1f5433-8dzrq"]
Oct 30 03:02:27.070028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:27.069998    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719f90bf-9cb3-4c6d-b802-c68c32b3c4e6" path="/var/lib/kubelet/pods/719f90bf-9cb3-4c6d-b802-c68c32b3c4e6/volumes"
Oct 30 03:02:28.041754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:28.041727    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:02:28.041937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:28.041801    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"]
Oct 30 03:02:28.050527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:28.050504    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6954229-xtmk4"]
Oct 30 03:02:29.071097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:29.071065    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9dc441-8bc0-421b-866f-f509478a57dd" path="/var/lib/kubelet/pods/0b9dc441-8bc0-421b-866f-f509478a57dd/volumes"
Oct 30 03:02:30.661019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.660989    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:02:30.665876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.665855    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.705940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.705799    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:02:30.760560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.760518    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.760751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.760571    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.760831 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.760761    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.760891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.760868    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.760973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.760957    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlp4j\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.761086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.761069    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.761141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.761121    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.761285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.761240    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.761380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.761359    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.869105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.868975    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.881984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.881954    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909324    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909495    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909582    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909665    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlp4j\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909770    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909824    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.912296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.909860    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.913008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.912973    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.914272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.913723    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.914438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.881737    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.914556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.882577    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.914657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.914174    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.914748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.914195    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.930549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.930520    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.945248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.940710    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.952380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.952352    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlp4j\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:30.993270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:30.988898    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:31.762313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:31.762283    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:02:31.767033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:31.766988    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:31.767215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:31.767109    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:31.767215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:31.767201    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:32.268098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.268027    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" event={"ID":"359e6cf9-4755-4460-b34b-7373de66a212","Type":"ContainerStarted","Data":"e903d457c550f39e346e34049c4545362b6b1ba3c3f05651c0c877bc1fe02837"}
Oct 30 03:02:32.268098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.268098    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" event={"ID":"359e6cf9-4755-4460-b34b-7373de66a212","Type":"ContainerStarted","Data":"9fa10bc5a21a51df6d76477dfdf097223b95f743b1945470061f34bbbf267886"}
Oct 30 03:02:32.324838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.324787    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podStartSLOduration=2.324772796 podStartE2EDuration="2.324772796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:32.323247984 +0000 UTC m=+1291.397418779" watchObservedRunningTime="2026-10-30 03:02:32.324772796 +0000 UTC m=+1291.398943588"
Oct 30 03:02:32.989429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.989395    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:32.992473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.992443    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:32.992583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:32.992486    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:33.032306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:33.032273    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"]
Oct 30 03:02:33.086914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:33.085040    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1b35eb-t48wn"]
Oct 30 03:02:33.989766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:33.989731    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:33.990225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:33.989788    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:34.989958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:34.989924    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:34.990433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:34.989996    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:35.170177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:35.170135    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c307a8-ac72-41f6-8cd4-234f1368933b" path="/var/lib/kubelet/pods/83c307a8-ac72-41f6-8cd4-234f1368933b/volumes"
Oct 30 03:02:36.009255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:36.008614    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:36.009255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:36.008679    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:36.992178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:36.992143    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:36.992473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:36.992446    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:37.991467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:37.991433    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:37.997425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:37.997382    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:38.431131 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.431088    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:02:38.458255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.456546    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.483266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.475413    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-be9f2a59-openshift-default-dockercfg-4gctp\""
Oct 30 03:02:38.504780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.504745    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:02:38.520259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.520217    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.520481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.520453    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.520655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.520631    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfcr8\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.520818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.520797    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.521061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.521039    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.521174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.521160    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.521341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.521324    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.521584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.521567    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.521731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.521704    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646450    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646503    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646547    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646704    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646741    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646837    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.646891    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.647015    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfcr8\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.647094    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.647738    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.647989    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.649253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.648221    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.666973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.666931    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.700309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.673103    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.742268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.716493    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.851986 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.851934    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.939046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.880994    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfcr8\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:38.947282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:38.945839    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:39.011102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:39.011018    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:39.012248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:39.012191    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:39.114599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:39.114566    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:39.285620 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:02:39.285534    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:02:40.312835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:40.262261    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:40.312835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:40.262594    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:40.973289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:40.971454    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:41.172454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.171968    4214 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:02:41.172454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.172023    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:02:41.174142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.174011    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:41.174383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.174359    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:41.174515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.174498    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:41.670262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.670212    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerStarted","Data":"3bfe6f3c63132a584a967b6cec9e0ea46810b4701dde4548fffae949590ad2c7"}
Oct 30 03:02:41.687265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:41.674315    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:02:42.665254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:42.664601    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerStarted","Data":"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"}
Oct 30 03:02:42.813254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:42.812716    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podStartSLOduration=4.81269394 podStartE2EDuration="4.81269394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:42.772560882 +0000 UTC m=+1301.846731676" watchObservedRunningTime="2026-10-30 03:02:42.81269394 +0000 UTC m=+1301.886864735"
Oct 30 03:02:42.872254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:42.867832    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:42.872254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:42.869649    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:42.890414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:42.887184    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:02:43.217255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:43.139891    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:02:43.217255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:43.139965    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:02:43.516305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:43.480938    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:43.874856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:43.874805    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.000758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:43.981503    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.025357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.023252    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.103268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.065660    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:02:44.103268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.065751    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.167331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.159035    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:44.447445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.447335    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:02:44.897494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.876152    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.929745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.929696    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:02:44.967216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:44.959550    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:02:45.142806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.142761    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.325220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.325174    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.488253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.487721    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:45.488253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.487785    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"]
Oct 30 03:02:45.488253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.487815    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde7b2fd-4bxkk"]
Oct 30 03:02:45.488253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.488207    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.519250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.490792    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:02:45.519250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.498635    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:02:45.519250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.504328    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:02:45.740804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.740720    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:02:46.000307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:45.975006    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:46.833598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:46.833545    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:47.168296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:47.144126    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60948ee3-4f25-4fc4-a82f-60353bba50ab" path="/var/lib/kubelet/pods/60948ee3-4f25-4fc4-a82f-60353bba50ab/volumes"
Oct 30 03:02:47.215252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:47.196083    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"]
Oct 30 03:02:47.215726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:47.215699    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed63bde4-bcnz7"]
Oct 30 03:02:49.087822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:49.087787    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae80dad1-fc2d-4271-b99a-babf572f1bb9" path="/var/lib/kubelet/pods/ae80dad1-fc2d-4271-b99a-babf572f1bb9/volumes"
Oct 30 03:02:51.054266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:51.054217    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"]
Oct 30 03:02:51.088270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:51.088222    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"]
Oct 30 03:02:51.093594 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:02:51.093556    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-963ff370-m5t54\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="cb542574-187d-48ec-94a3-2241ea14b3b8" pod="odh-ai-gateway-infra/maas-api-revoke-sub-963ff370-m5t54"
Oct 30 03:02:52.491376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.491337    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:02:52.496375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.496343    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:52.522726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.522694    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:02:52.526211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.526175    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsshm\" (UniqueName: \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:52.526722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.526692    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:52.583995 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.576675    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-723ffc-dockercfg-8cr5m\""
Oct 30 03:02:52.611251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.595527    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-723ffc\""
Oct 30 03:02:52.691264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.685202    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:52.691264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.685276    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:52.691264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:52.685618    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsshm\" (UniqueName: \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:53.091251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:53.090697    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsshm\" (UniqueName: \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\") pod \"payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:53.117714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:53.117683    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:53.164503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:53.164459    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb542574-187d-48ec-94a3-2241ea14b3b8" path="/var/lib/kubelet/pods/cb542574-187d-48ec-94a3-2241ea14b3b8/volumes"
Oct 30 03:02:53.557965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:53.557929    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:02:54.175043 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:54.174998    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" event={"ID":"8736cfd8-158b-407c-a473-7755bbb10ce2","Type":"ContainerStarted","Data":"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"}
Oct 30 03:02:54.175333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:54.175068    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" event={"ID":"8736cfd8-158b-407c-a473-7755bbb10ce2","Type":"ContainerStarted","Data":"8e2017ba72ae818add58eb8159c0f793545b307bfd930c8d8d363b1257cd6b14"}
Oct 30 03:02:54.175333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:54.175099    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:02:54.223277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:54.223203    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" podStartSLOduration=2.22318948 podStartE2EDuration="2.22318948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:54.21688618 +0000 UTC m=+1313.291056974" watchObservedRunningTime="2026-10-30 03:02:54.22318948 +0000 UTC m=+1313.297360272"
Oct 30 03:02:56.077921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:56.077853    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"]
Oct 30 03:02:56.102664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:56.102617    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527922ba-s6b66"]
Oct 30 03:02:57.073000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:57.072961    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca90fa5-4f50-4699-bb74-064514822211" path="/var/lib/kubelet/pods/5ca90fa5-4f50-4699-bb74-064514822211/volumes"
Oct 30 03:02:58.063489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.063453    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:02:58.064127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.064091    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:02:58.669726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.668520    4214 generic.go:356] "Generic (PLEG): container finished" podID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerID="e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e" exitCode=0
Oct 30 03:02:58.669726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.668591    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerDied","Data":"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"}
Oct 30 03:02:58.669726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.669093    4214 scope.go:122] "RemoveContainer" containerID="e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"
Oct 30 03:02:58.671419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.671310    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:58.671419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.671384    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:58.671419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:58.671416    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.106257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.103824    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"]
Oct 30 03:02:59.143523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.141448    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"]
Oct 30 03:02:59.143523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.141566    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205209    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205288    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e682e6-c3a8-434f-9fae-06137c443148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205442    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205486    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205572    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205655    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e682e6-c3a8-434f-9fae-06137c443148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205687    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkpd\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-kube-api-access-phkpd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205854    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.205941 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.205889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.307932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.307888    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e682e6-c3a8-434f-9fae-06137c443148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.307952    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkpd\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-kube-api-access-phkpd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308089    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308116    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308277    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308334    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e682e6-c3a8-434f-9fae-06137c443148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308473    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308491    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308526    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308557    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308645    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.308987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.308964    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.309214 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.309189    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.309360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.309217    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e682e6-c3a8-434f-9fae-06137c443148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.311074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.310999    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e682e6-c3a8-434f-9fae-06137c443148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.311074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.311057    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e682e6-c3a8-434f-9fae-06137c443148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.331435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.331216    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkpd\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-kube-api-access-phkpd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.333305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.333163    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e682e6-c3a8-434f-9fae-06137c443148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xmv89\" (UID: \"f4e682e6-c3a8-434f-9fae-06137c443148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.462301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.462023    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:02:59.696876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.696843    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerStarted","Data":"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"}
Oct 30 03:02:59.767502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.767395    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:02:59.767502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.767459    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:02:59.767781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.767758    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:02:59.873976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.873904    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.875141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.875094    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.875390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.875354    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:02:59.888265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:02:59.888215    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"]
Oct 30 03:03:00.762454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:00.757590    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:00.762454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:00.757661    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:00.800726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:00.800679    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" event={"ID":"f4e682e6-c3a8-434f-9fae-06137c443148","Type":"ContainerStarted","Data":"79d1c1745e21dd93395c71bf4743bc8520288e26d268a803bbb05143721bad30"}
Oct 30 03:03:00.800929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:00.800746    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" event={"ID":"f4e682e6-c3a8-434f-9fae-06137c443148","Type":"ContainerStarted","Data":"3ce0557c1bb147275d9af6569d46da03ca54aaa031d72c52652c974ed6665a62"}
Oct 30 03:03:00.900779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:00.899147    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podStartSLOduration=1.8991325479999999 podStartE2EDuration="1.899132548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:00.885730427 +0000 UTC m=+1319.959901211" watchObservedRunningTime="2026-10-30 03:03:00.899132548 +0000 UTC m=+1319.973303342"
Oct 30 03:03:01.469251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:01.468919    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:03:01.469251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:01.469062    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:01.469251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:01.469106    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:01.822393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:01.822357    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:01.822980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:01.822938    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:02.652312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:02.652273    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:02.698956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:02.698495    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:02.785253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:02.781333    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:02.785253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:02.781768    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:02.870075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:02.868896    4214 scope.go:122] "RemoveContainer" containerID="9a466676062273900d42ec2a1ee645d0e4712b8224ae899818a76c47c19c65a5"
Oct 30 03:03:03.133405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:03.133350    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:03:03.468256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:03.463618    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:03.468256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:03.463696    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:03.758455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:03.758356    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:03.758455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:03.758427    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:04.487757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:04.484535    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:04.487757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:04.484616    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:04.758313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:04.757652    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:04.758313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:04.757731    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:05.195950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:05.195920    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:03:05.465316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:05.464642    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:05.465316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:05.464709    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:05.760089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:05.759994    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 03:03:05.760699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:05.760666    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 03:03:06.214339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.214301    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"]
Oct 30 03:03:06.307296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.307191    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b3c61a-hzqhs"]
Oct 30 03:03:06.469507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.469395    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:06.469507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.469483    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:06.769731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.769651    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:03:06.771023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.771005    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:03:06.771775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:06.771756    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:03:07.077166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:07.077124    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3018a4-a1be-4bc3-a528-c0e27a2329b2" path="/var/lib/kubelet/pods/5b3018a4-a1be-4bc3-a528-c0e27a2329b2/volumes"
Oct 30 03:03:07.465736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:07.465646    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:07.465892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:07.465729    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:08.463794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:08.463756    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:08.464182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:08.463822    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:09.169123 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:09.169088    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:03:09.463021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:09.462911    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 03:03:09.463021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:09.462986    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 03:03:10.693466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:10.693429    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:03:10.693958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:10.693933    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:03:10.695221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:10.695196    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89"
Oct 30 03:03:11.076983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.076947    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:03:11.084315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.084289    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.095304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.095274    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:03:11.100577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.100548    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-be9f2a59-serving-cert\""
Oct 30 03:03:11.193713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.193676    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.193713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.193720    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.193998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.193900    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8zc\" (UniqueName: \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.194122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.194099    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.304643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.304602    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.304947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.304918    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.305263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.305223    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8zc\" (UniqueName: \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.305730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.305682    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.314524 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:11.314495    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-serving-cert: secret "maas-api-e2e-mt-be9f2a59-serving-cert" not found
Oct 30 03:03:11.314928 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:11.314908    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls podName:1b4b021c-f644-47e4-b898-c8af7b36a808 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:11.814878453 +0000 UTC m=+1330.889049241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls") pod "maas-api-e2e-mt-be9f2a59-697585b688-96wfw" (UID: "1b4b021c-f644-47e4-b898-c8af7b36a808") : secret "maas-api-e2e-mt-be9f2a59-serving-cert" not found
Oct 30 03:03:11.317295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.317268    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.474309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.445119    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8zc\" (UniqueName: \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.474309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.450879    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.861878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.861842    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:11.864445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:11.864415    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-96wfw\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:12.011278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:12.009715    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:12.385840 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:03:12.385776    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b021c_f644_47e4_b898_c8af7b36a808.slice/crio-8997a11b685e88ecd6665e67f71e129ce03310cae4c3244c9d5d0b177012e3b3 WatchSource:0}: Error finding container 8997a11b685e88ecd6665e67f71e129ce03310cae4c3244c9d5d0b177012e3b3: Status 404 returned error can't find the container with id 8997a11b685e88ecd6665e67f71e129ce03310cae4c3244c9d5d0b177012e3b3
Oct 30 03:03:12.392642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:12.392614    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:03:12.398591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:12.398558    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" event={"ID":"1b4b021c-f644-47e4-b898-c8af7b36a808","Type":"ContainerStarted","Data":"8997a11b685e88ecd6665e67f71e129ce03310cae4c3244c9d5d0b177012e3b3"}
Oct 30 03:03:13.461251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:13.454670    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" event={"ID":"1b4b021c-f644-47e4-b898-c8af7b36a808","Type":"ContainerStarted","Data":"f500f1fdbca8dc70eaeb911e87b878708b29a8843613e431fd7cab2ef3e4eff9"}
Oct 30 03:03:13.461251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:13.456899    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:13.519024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:13.518959    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" podStartSLOduration=2.103626466 podStartE2EDuration="2.518930448s" totalImagesPullingTime="415.303982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:13.510251107 +0000 UTC m=+1332.584421904" watchObservedRunningTime="2026-10-30 03:03:13.518930448 +0000 UTC m=+1332.593101246"
Oct 30 03:03:15.388308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.387393    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:03:15.391549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.391525    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.407120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.407095    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:03:15.413864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.413831    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-723ffc-serving-cert\""
Oct 30 03:03:15.443675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.442584    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.443967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.443944    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.444153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.444137    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.444313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.444296    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2p28\" (UniqueName: \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.546820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.546782    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.547036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.546886    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.547036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.546959    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.547175 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:15.547035    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-serving-cert: secret "maas-api-e2e-shared-a-w3-723ffc-serving-cert" not found
Oct 30 03:03:15.547175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.547060    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2p28\" (UniqueName: \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.547175 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:15.547113    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls podName:014aa7e5-12d6-49e4-b052-adc27bc226a4 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:16.047092319 +0000 UTC m=+1335.121263093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" (UID: "014aa7e5-12d6-49e4-b052-adc27bc226a4") : secret "maas-api-e2e-shared-a-w3-723ffc-serving-cert" not found
Oct 30 03:03:15.547833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.547787    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.550221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.550190    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:15.569038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:15.569001    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2p28\" (UniqueName: \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:16.036559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.036521    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"]
Oct 30 03:03:16.042341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.042308    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d0786f0-5rftw"]
Oct 30 03:03:16.064453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.064417    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:16.067288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.067264    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:16.318992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.318482    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:16.830830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:16.827868    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:03:17.073976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:17.073933    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85c827b-3ea4-41d0-8d57-83b658e9d836" path="/var/lib/kubelet/pods/e85c827b-3ea4-41d0-8d57-83b658e9d836/volumes"
Oct 30 03:03:17.513093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:17.512997    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" event={"ID":"014aa7e5-12d6-49e4-b052-adc27bc226a4","Type":"ContainerStarted","Data":"f87b2aa220743c7c8c1ca79d454b30ba453fa492c47a9606c3a59e8eed7f88f6"}
Oct 30 03:03:17.513093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:17.513068    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" event={"ID":"014aa7e5-12d6-49e4-b052-adc27bc226a4","Type":"ContainerStarted","Data":"32951a7c326374f8c12a4e9892aacc0cfd13e73ed29d3804c443af517d0f7ee4"}
Oct 30 03:03:17.513093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:17.513090    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:17.557252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:17.556828    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" podStartSLOduration=2.115084555 podStartE2EDuration="2.556811004s" totalImagesPullingTime="441.726449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:17.555827197 +0000 UTC m=+1336.629997990" watchObservedRunningTime="2026-10-30 03:03:17.556811004 +0000 UTC m=+1336.630981800"
Oct 30 03:03:20.683647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:20.683609    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:03:23.032953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.032919    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"]
Oct 30 03:03:23.046687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.046650    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f31944-4gcns"]
Oct 30 03:03:23.073474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.073437    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b" path="/var/lib/kubelet/pods/fd8c7de3-d1f3-4b3e-ad27-2e76a5f85a0b/volumes"
Oct 30 03:03:23.145404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.145366    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:03:23.150378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.150356    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.166272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.166213    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-ae1dd9-serving-cert\""
Oct 30 03:03:23.171442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.171414    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:03:23.243164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.243118    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.243432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.243407    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474c7\" (UniqueName: \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.243559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.243533    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.243659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.243638    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.345346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.345312    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.345557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.345538    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474c7\" (UniqueName: \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.345648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.345632    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.345729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.345713    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.346456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.346432    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.348177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.348151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.348392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.348371    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.411712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.399656    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-474c7\" (UniqueName: \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\") pod \"maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.468031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.467992    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:23.567329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.566756    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:03:23.917078 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:03:23.917045    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f0cdb8_ec94_4a1f_a68e_80ee33db0a50.slice/crio-a4a1df38b6e749d2d93e5b664a15d7cfb7542a67a315cab58f14b5d57672d45e WatchSource:0}: Error finding container a4a1df38b6e749d2d93e5b664a15d7cfb7542a67a315cab58f14b5d57672d45e: Status 404 returned error can't find the container with id a4a1df38b6e749d2d93e5b664a15d7cfb7542a67a315cab58f14b5d57672d45e
Oct 30 03:03:23.918699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:23.918672    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:03:24.854758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:24.854651    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" event={"ID":"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50","Type":"ContainerStarted","Data":"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"}
Oct 30 03:03:24.854758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:24.854722    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" event={"ID":"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50","Type":"ContainerStarted","Data":"a4a1df38b6e749d2d93e5b664a15d7cfb7542a67a315cab58f14b5d57672d45e"}
Oct 30 03:03:24.854758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:24.854751    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:24.901777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:24.901718    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" podStartSLOduration=1.509839403 podStartE2EDuration="1.90170437s" totalImagesPullingTime="391.864967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:24.897749573 +0000 UTC m=+1343.971920369" watchObservedRunningTime="2026-10-30 03:03:24.90170437 +0000 UTC m=+1343.975875196"
Oct 30 03:03:30.871531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:30.871499    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:03:32.033508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:32.033467    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"]
Oct 30 03:03:32.050383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:32.050339    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6809c4-p99wp"]
Oct 30 03:03:33.073355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:33.073318    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc25c04-cb04-4ca3-a7a1-466b98bf9763" path="/var/lib/kubelet/pods/8cc25c04-cb04-4ca3-a7a1-466b98bf9763/volumes"
Oct 30 03:03:37.030139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:37.030100    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"]
Oct 30 03:03:37.045650 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:37.045617    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e20c6-mzcpc"]
Oct 30 03:03:37.073496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:37.073459    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76b8c43-53fb-43a1-93dc-8be38e7e3062" path="/var/lib/kubelet/pods/e76b8c43-53fb-43a1-93dc-8be38e7e3062/volumes"
Oct 30 03:03:39.173992 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:03:39.173953    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:03:47.175199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:47.175167    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"]
Oct 30 03:03:47.175867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:47.175848    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8d358ab-sw6d9"]
Oct 30 03:03:49.085599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:49.085563    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec176c05-9c6b-4e24-9ac2-d3a4c586b72e" path="/var/lib/kubelet/pods/ec176c05-9c6b-4e24-9ac2-d3a4c586b72e/volumes"
Oct 30 03:03:50.872609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:50.872550    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:03:50.879016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:50.878984    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:50.893662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:50.893632    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:03:50.932178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:50.923523    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ffb1886-dockercfg-lqbr5\""
Oct 30 03:03:50.939253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:50.931447    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ffb1886\""
Oct 30 03:03:51.022255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.011737    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.022255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.012038    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tks2\" (UniqueName: \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.153541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.144375    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tks2\" (UniqueName: \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.153541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.144676    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.217255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.209098    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.330434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.330358    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tks2\" (UniqueName: \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\") pod \"payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.545726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.544298    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:51.937384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:51.937295    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:03:52.101768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.100214    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.128048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.128004    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8e2727-openshift-default-dockercfg-c6drp\""
Oct 30 03:03:52.352301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.348784    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.352301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.348967    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.352301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.349025    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.352301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.349095    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.352301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.349191    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.382486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.381586    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.382486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.381637    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbt9\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.382486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.381766    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.382486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.381815    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.398110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.397987    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" event={"ID":"7381385e-3142-47e0-a688-2ca263b966c9","Type":"ContainerStarted","Data":"8fa57b942ea759f156d54892b93b6b7900fde3c4d3186318df9270555cfe2486"}
Oct 30 03:03:52.471555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.471510    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:03:52.471913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.471892    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:03:52.485361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.485260    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.486177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.486147    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.486432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.486414    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.486618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.486602    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.486833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.486814    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.487138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.487119    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.487432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.487402    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbt9\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.487680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.487655    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.487853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.487823    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.488980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.488946    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.518089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.516476    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.518089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.516772    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.518089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.517019    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.518089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.517273    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.551761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.551720    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.619251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.603837    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbt9\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.629743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.628799    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.708030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.706818    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\") pod \"e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") " pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:52.948513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:52.948480    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:53.504697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.496610    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" event={"ID":"7381385e-3142-47e0-a688-2ca263b966c9","Type":"ContainerStarted","Data":"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"}
Oct 30 03:03:53.504697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.496815    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:03:53.562634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.562569    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" podStartSLOduration=3.562549915 podStartE2EDuration="3.562549915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:53.557876061 +0000 UTC m=+1372.632046867" watchObservedRunningTime="2026-10-30 03:03:53.562549915 +0000 UTC m=+1372.636720709"
Oct 30 03:03:53.578901 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:03:53.578865    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ce25d3_ead3_45dc_b5d4_fa157d141a01.slice/crio-9c70c74bb517d8dc8621d89c3b2347b56d3a6865df0cd10db5ff5625eaa413af WatchSource:0}: Error finding container 9c70c74bb517d8dc8621d89c3b2347b56d3a6865df0cd10db5ff5625eaa413af: Status 404 returned error can't find the container with id 9c70c74bb517d8dc8621d89c3b2347b56d3a6865df0cd10db5ff5625eaa413af
Oct 30 03:03:53.581672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.581640    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:03:53.583004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.582966    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:53.583117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.583052    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:53.583117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:53.583095    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:54.510700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:54.510655    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerStarted","Data":"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"}
Oct 30 03:03:54.511185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:54.510726    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerStarted","Data":"9c70c74bb517d8dc8621d89c3b2347b56d3a6865df0cd10db5ff5625eaa413af"}
Oct 30 03:03:54.581837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:54.581765    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podStartSLOduration=3.5817486179999998 podStartE2EDuration="3.581748618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:54.567183317 +0000 UTC m=+1373.641354112" watchObservedRunningTime="2026-10-30 03:03:54.581748618 +0000 UTC m=+1373.655919410"
Oct 30 03:03:54.978300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:54.975785    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:54.987144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:54.987108    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:55.555222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:55.543201    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:55.555222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:55.545472    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:03:57.693422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.693372    4214 generic.go:356] "Generic (PLEG): container finished" podID="ce74a684-df53-4681-94e8-01802098f1b4" containerID="708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5" exitCode=0
Oct 30 03:03:57.695869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.695827    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerDied","Data":"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"}
Oct 30 03:03:57.709143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.709110    4214 scope.go:122] "RemoveContainer" containerID="708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"
Oct 30 03:03:57.717267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.717215    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:57.717515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.717494    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:57.717654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:57.717638    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:03:58.714364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:58.714320    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerStarted","Data":"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"}
Oct 30 03:03:59.122380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:59.122346    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:03:59.122751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:59.122720    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:03:59.122928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:03:59.122901    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:00.075420 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:00.075371    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"]
Oct 30 03:04:00.099254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:00.099205    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe0cffc-2xzg6"]
Oct 30 03:04:00.116955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:00.116921    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:00.117112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:00.116978    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:00.312433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:00.312400    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:04:01.084898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:01.084863    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43afc460-e3bd-4527-aade-e879b6b02695" path="/var/lib/kubelet/pods/43afc460-e3bd-4527-aade-e879b6b02695/volumes"
Oct 30 03:04:01.116084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:01.115906    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:01.116084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:01.115986    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:02.228168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:02.228128    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:02.228742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:02.228185    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:03.116106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.116023    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:03.116106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.116070    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:03.212624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.212589    4214 scope.go:122] "RemoveContainer" containerID="3f92d3f4786c20d121ad79d1711d5a13f045d93358d201f192e748d2c345a712"
Oct 30 03:04:03.224816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.224795    4214 scope.go:122] "RemoveContainer" containerID="65b94f994a485670255a7694cdd91a62d254b4a339223ca962b74e0594ef22e5"
Oct 30 03:04:03.280731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.280700    4214 scope.go:122] "RemoveContainer" containerID="43d06bf77ff6a1dfb51828fdf2b7d504a9e373dcf9fea1df9482c7cc5148cc65"
Oct 30 03:04:03.302336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.302311    4214 scope.go:122] "RemoveContainer" containerID="133aa75f69fc44a970a0a59d16888f8d29bb6a695b61150ef2821d319a5fb53d"
Oct 30 03:04:03.315586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.315562    4214 scope.go:122] "RemoveContainer" containerID="ab4c9a61f2c56ab8b7262c2ad75e65acd2abd8a1e83c914468bbc6a64358c820"
Oct 30 03:04:03.326820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.326797    4214 scope.go:122] "RemoveContainer" containerID="e1f7152e1a8149794afd73ae5f479ec0e47db68de2d8ebca855cd53a9567e0ba"
Oct 30 03:04:03.368510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.368473    4214 scope.go:122] "RemoveContainer" containerID="52bdd2b9d447a4871af47d6aca8ed7d3a2397a713655b50802ba00e2405b88e0"
Oct 30 03:04:03.387109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:03.387085    4214 scope.go:122] "RemoveContainer" containerID="0f38a417a4cba557a736611a1c147b807d262d5aff5b0e247638e0eb2ad004c8"
Oct 30 03:04:04.088085 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:04.087916    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"]
Oct 30 03:04:04.161317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:04.161219    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:04.163768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:04.163732    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:04.165785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:04.165728    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2015db-c64kb"]
Oct 30 03:04:04.513946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:04.513872    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:04:05.064707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:05.059848    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"]
Oct 30 03:04:05.116132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:05.115719    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:04:05.116132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:05.115777    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 03:04:05.138253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:05.132970    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a383e0-4b39-4356-a4fa-2678a8f5de0a" path="/var/lib/kubelet/pods/23a383e0-4b39-4356-a4fa-2678a8f5de0a/volumes"
Oct 30 03:04:05.203028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:05.202978    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743c3e8c-zrjsn"]
Oct 30 03:04:06.144634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:06.144601    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:04:06.162253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:06.160801    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:04:07.070113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:07.070081    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1cc68a-0765-4bec-a2b0-623de738cfef" path="/var/lib/kubelet/pods/0d1cc68a-0765-4bec-a2b0-623de738cfef/volumes"
Oct 30 03:04:09.168847 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:09.168806    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:04:10.143794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:10.143479    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"]
Oct 30 03:04:10.176414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:10.166443    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34292810-vrt8c"]
Oct 30 03:04:11.267381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:11.267274    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562212ed-d68e-48af-ad4c-e784e3c07859" path="/var/lib/kubelet/pods/562212ed-d68e-48af-ad4c-e784e3c07859/volumes"
Oct 30 03:04:13.039979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:13.039938    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"]
Oct 30 03:04:13.051147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:13.051117    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2556782-9smwd"]
Oct 30 03:04:13.069545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:13.069508    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d4f09b-29dd-45e3-be9a-60d4771bc838" path="/var/lib/kubelet/pods/09d4f09b-29dd-45e3-be9a-60d4771bc838/volumes"
Oct 30 03:04:16.032729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:16.032647    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"]
Oct 30 03:04:16.045152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:16.045111    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05266894-5v9px"]
Oct 30 03:04:17.078469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:17.078433    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f072ce6-bff1-4b1c-bc0a-c23798419ebe" path="/var/lib/kubelet/pods/7f072ce6-bff1-4b1c-bc0a-c23798419ebe/volumes"
Oct 30 03:04:25.036460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.036425    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:04:25.360995 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.360087    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.568421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.554202    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:04:25.805251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.794767    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.805251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.794928    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bqs\" (UniqueName: \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.810947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.810908    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:04:25.824275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.814049    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:04:25.831250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.827504    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d30e9f\""
Oct 30 03:04:25.831250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.829300    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d30e9f-dockercfg-j64ks\""
Oct 30 03:04:25.896915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.896828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bqs\" (UniqueName: \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.982347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.979798    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.982347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.980916    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:25.982347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:25.975547    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:04:26.000202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:26.000165    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:04:26.043203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:26.041835    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bqs\" (UniqueName: \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\") pod \"payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:26.072386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:26.054264    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:27.028269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:27.023450    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:04:27.912177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:27.912140    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" event={"ID":"86020f30-a736-4491-88b8-2ca91aaa570f","Type":"ContainerStarted","Data":"6f6ced2f503e5f631a25439eef554c6e6304807947b5913241a10e479f3793a4"}
Oct 30 03:04:27.912671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:27.912217    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:27.912671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:27.912260    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" event={"ID":"86020f30-a736-4491-88b8-2ca91aaa570f","Type":"ContainerStarted","Data":"a838c0ba44b8c7b06bcabed3e07fa64528f730369190d1b61ce022cece5d7291"}
Oct 30 03:04:27.955396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:27.955351    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" podStartSLOduration=3.9553370450000003 podStartE2EDuration="3.955337045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:27.951545482 +0000 UTC m=+1407.025716275" watchObservedRunningTime="2026-10-30 03:04:27.955337045 +0000 UTC m=+1407.029507839"
Oct 30 03:04:38.984280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:38.984211    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:04:39.045580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:39.045543    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"]
Oct 30 03:04:39.054783 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:39.054750    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf58e08-glghw"]
Oct 30 03:04:39.089838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:39.089798    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e4d8f4-da3e-4526-951a-bcc0aa4cd72e" path="/var/lib/kubelet/pods/13e4d8f4-da3e-4526-951a-bcc0aa4cd72e/volumes"
Oct 30 03:04:39.196904 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:39.196869    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:04:45.859114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.859078    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 03:04:45.859688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.859331    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy" containerID="cri-o://5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629" gracePeriod=30
Oct 30 03:04:45.871863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.871829    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 03:04:45.872111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.872066    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy" containerID="cri-o://bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20" gracePeriod=30
Oct 30 03:04:45.880934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.880908    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 03:04:45.881348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.881316    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy" containerID="cri-o://1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659" gracePeriod=30
Oct 30 03:04:45.887000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.886965    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 03:04:45.887265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:45.887223    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" containerID="cri-o://c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479" gracePeriod=30
Oct 30 03:04:48.962976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:48.962926    4214 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:04:48.963442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:48.963001    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:04:51.241635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.241599    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 03:04:51.354346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.354264    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:04:51.356325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.355795    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy"
Oct 30 03:04:51.356325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.355823    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy"
Oct 30 03:04:51.356325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.356108    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerName="istio-proxy"
Oct 30 03:04:51.361899 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.361479    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.368709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.368682    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.376540    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.376933    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377035    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377066    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377129    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377159    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377373    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377527    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377691    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377735    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377816    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\") pod \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\" (UID: \"cb1d97f4-1c98-4fce-951b-ac3de38bf081\") "
Oct 30 03:04:51.377937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.377864    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.378613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378357    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:04:51.378613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378587    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.378889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378866    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.378956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378895    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.378956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378910    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.378956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378924    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.378956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.378937    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.380856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.380801    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "kube-api-access-vcfl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.381540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.381256    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.381540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.381439    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1abc2-dockercfg-d5z5k\""
Oct 30 03:04:51.382265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.382203    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.382474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.382455    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo" pod "cb1d97f4-1c98-4fce-951b-ac3de38bf081" (UID: "cb1d97f4-1c98-4fce-951b-ac3de38bf081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:04:51.415558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.393548    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 03:04:51.415558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.404450    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 03:04:51.415558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.410761    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 03:04:51.443251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.438061    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1abc2\""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506407    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506484    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506721    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506769    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506805    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506929    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.506996    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.507064    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.507102    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\") pod \"09d30799-48f2-4909-841b-cb2f91fffd0e\" (UID: \"09d30799-48f2-4909-841b-cb2f91fffd0e\") "
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.507613    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wn5m\" (UniqueName: \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.507681    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.508288    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfl4\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-kube-api-access-vcfl4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.508311    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.508327    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.508340    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1d97f4-1c98-4fce-951b-ac3de38bf081-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.539293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.539653    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.540066    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.543102    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.543302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.543188    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:04:51.603321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.556565    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.603321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.597350    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.604221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.604146    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "kube-api-access-sh744". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.605624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.605592    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy" pod "09d30799-48f2-4909-841b-cb2f91fffd0e" (UID: "09d30799-48f2-4909-841b-cb2f91fffd0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609095    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609149    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609167    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609186    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609221    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609275    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609292    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609334    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609432    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609465    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609545    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609563    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609602    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609653    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\") pod \"432ca094-9b1c-4d92-9223-91f93b9bb441\" (UID: \"432ca094-9b1c-4d92-9223-91f93b9bb441\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609807    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609860    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609894    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.609951    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\") pod \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\" (UID: \"5e774eaf-0d2a-462f-ae16-ad0e86be3496\") "
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.610798    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wn5m\" (UniqueName: \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.610865    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611272    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611302    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611317    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh744\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-kube-api-access-sh744\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611331    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611347    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611360    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d30799-48f2-4909-841b-cb2f91fffd0e-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611372    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611384    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d30799-48f2-4909-841b-cb2f91fffd0e-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.612302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.611397    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d30799-48f2-4909-841b-cb2f91fffd0e-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.613826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.613120    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.614607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.614311    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.620353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.620322    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.621318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.621138    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:04:51.621924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.621901    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.622327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.622304    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.623053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.623007    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.630698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.629595    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "kube-api-access-spqjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.630698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.630437    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.632275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.630953    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.632275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.631025    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632501    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632523    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632565    4214 cpu_manager.go:414] "containerMap: removing container" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632574    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632592    4214 cpu_manager.go:414] "containerMap: removing container" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632599    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632617    4214 cpu_manager.go:414] "containerMap: removing container" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632624    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632874    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632888    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632898    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerName="istio-proxy"
Oct 30 03:04:51.635249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.632910    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerName="istio-proxy"
Oct 30 03:04:51.637939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.637914    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.645340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.644958    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:04:51.645451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.645393    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.645706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.645608    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.645706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.645658    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:04:51.645706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.645660    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:04:51.651567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.651287    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wn5m\" (UniqueName: \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\") pod \"payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.654967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.654942    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-44a8fd-serving-cert\""
Oct 30 03:04:51.661358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.661332    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:04:51.661764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.661724    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:04:51.661764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.661731    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "kube-api-access-nk5f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.661916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.661859    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token" pod "432ca094-9b1c-4d92-9223-91f93b9bb441" (UID: "432ca094-9b1c-4d92-9223-91f93b9bb441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.663749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.663722    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token" pod "5e774eaf-0d2a-462f-ae16-ad0e86be3496" (UID: "5e774eaf-0d2a-462f-ae16-ad0e86be3496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:04:51.700254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.698555    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:51.719657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719285    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8f6\" (UniqueName: \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.719657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719379    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.719657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719496    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.719657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719577    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.719977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719948    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.719977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719971    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719986    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.719998    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720010    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720023    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720035    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720048    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720060    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432ca094-9b1c-4d92-9223-91f93b9bb441-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720073    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720086    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720100    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5f2\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-kube-api-access-nk5f2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720115    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432ca094-9b1c-4d92-9223-91f93b9bb441-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720127    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720142    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720155    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqjz\" (UniqueName: \"kubernetes.io/projected/5e774eaf-0d2a-462f-ae16-ad0e86be3496-kube-api-access-spqjz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720169    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e774eaf-0d2a-462f-ae16-ad0e86be3496-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.720575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.720182    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432ca094-9b1c-4d92-9223-91f93b9bb441-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:04:51.757366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.757223    4214 generic.go:356] "Generic (PLEG): container finished" podID="09d30799-48f2-4909-841b-cb2f91fffd0e" containerID="bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20" exitCode=0
Oct 30 03:04:51.757532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.757379    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" event={"ID":"09d30799-48f2-4909-841b-cb2f91fffd0e","Type":"ContainerDied","Data":"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"}
Oct 30 03:04:51.757532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.757477    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c" event={"ID":"09d30799-48f2-4909-841b-cb2f91fffd0e","Type":"ContainerDied","Data":"237cacbb900f06691ad028d14844964ddeb19e9773f1b604abd26ef636466c5e"}
Oct 30 03:04:51.757532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.757396    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"
Oct 30 03:04:51.757756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.757575    4214 scope.go:122] "RemoveContainer" containerID="bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"
Oct 30 03:04:51.760257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.759568    4214 generic.go:356] "Generic (PLEG): container finished" podID="432ca094-9b1c-4d92-9223-91f93b9bb441" containerID="c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479" exitCode=0
Oct 30 03:04:51.760257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.759671    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"
Oct 30 03:04:51.760257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.759744    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerDied","Data":"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"}
Oct 30 03:04:51.760257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.759790    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s" event={"ID":"432ca094-9b1c-4d92-9223-91f93b9bb441","Type":"ContainerDied","Data":"364df988e722d9e2c4840f779b705788111bb5a6f6255061e3b42b771c5847b5"}
Oct 30 03:04:51.766918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.766878    4214 generic.go:356] "Generic (PLEG): container finished" podID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" containerID="5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629" exitCode=0
Oct 30 03:04:51.767106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.766974    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"
Oct 30 03:04:51.767106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.767088    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" event={"ID":"5e774eaf-0d2a-462f-ae16-ad0e86be3496","Type":"ContainerDied","Data":"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"}
Oct 30 03:04:51.767283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.767135    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc" event={"ID":"5e774eaf-0d2a-462f-ae16-ad0e86be3496","Type":"ContainerDied","Data":"e21115e4e06ac410325fdae58b3e1e57a5d490f9da8dd6e01630d5c7d4d7b3f3"}
Oct 30 03:04:51.769095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.769068    4214 generic.go:356] "Generic (PLEG): container finished" podID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" containerID="1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659" exitCode=0
Oct 30 03:04:51.769198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.769168    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" event={"ID":"cb1d97f4-1c98-4fce-951b-ac3de38bf081","Type":"ContainerDied","Data":"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"}
Oct 30 03:04:51.769308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.769204    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4" event={"ID":"cb1d97f4-1c98-4fce-951b-ac3de38bf081","Type":"ContainerDied","Data":"c31f0e5afe51671a58c804c1fc62abf6742c8a04cc021a2d033a1f319b99b1ad"}
Oct 30 03:04:51.769308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.769302    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"
Oct 30 03:04:51.783669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.783648    4214 scope.go:122] "RemoveContainer" containerID="bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"
Oct 30 03:04:51.784246 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:51.784097    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20\": container with ID starting with bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20 not found: ID does not exist" containerID="bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"
Oct 30 03:04:51.784246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.784131    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20"} err="failed to get container status \"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20\": rpc error: code = NotFound desc = could not find container \"bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20\": container with ID starting with bbeee7fbe949c680129fa302d0dc1dcddf4b9a93f661587ebbcb3733e0fa5f20 not found: ID does not exist"
Oct 30 03:04:51.784246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.784156    4214 scope.go:122] "RemoveContainer" containerID="c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"
Oct 30 03:04:51.799881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.799853    4214 scope.go:122] "RemoveContainer" containerID="e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"
Oct 30 03:04:51.820405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.820383    4214 scope.go:122] "RemoveContainer" containerID="c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"
Oct 30 03:04:51.825401 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:51.825371    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479\": container with ID starting with c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479 not found: ID does not exist" containerID="c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"
Oct 30 03:04:51.825513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.825413    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479"} err="failed to get container status \"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479\": rpc error: code = NotFound desc = could not find container \"c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479\": container with ID starting with c4a4cc314c3b95eac236363c87dca74f8c55ffb7017aa94d52e43037d5d2d479 not found: ID does not exist"
Oct 30 03:04:51.825513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.825436    4214 scope.go:122] "RemoveContainer" containerID="e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"
Oct 30 03:04:51.826646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.826620    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8f6\" (UniqueName: \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.826752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.826675    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.826820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.826778    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.826876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.826856    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.827725 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:51.827691    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b\": container with ID starting with e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b not found: ID does not exist" containerID="e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"
Oct 30 03:04:51.827823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.827730    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b"} err="failed to get container status \"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b\": rpc error: code = NotFound desc = could not find container \"e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b\": container with ID starting with e5d3db43979a5ec86832092d444b9df49f1d2d8e0e22c8260f8a983a8487a42b not found: ID does not exist"
Oct 30 03:04:51.827823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.827754    4214 scope.go:122] "RemoveContainer" containerID="5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"
Oct 30 03:04:51.827823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.827757    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.830575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.830505    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.831708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.831686    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.843590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.843500    4214 scope.go:122] "RemoveContainer" containerID="5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"
Oct 30 03:04:51.843821 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:51.843798    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629\": container with ID starting with 5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629 not found: ID does not exist" containerID="5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"
Oct 30 03:04:51.843902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.843830    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629"} err="failed to get container status \"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629\": rpc error: code = NotFound desc = could not find container \"5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629\": container with ID starting with 5c1398e84e04aebf26c888b93fe8cf21443b5fc252c812ca43ae65058fdea629 not found: ID does not exist"
Oct 30 03:04:51.843902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.843851    4214 scope.go:122] "RemoveContainer" containerID="1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"
Oct 30 03:04:51.853974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.853950    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8f6\" (UniqueName: \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\") pod \"maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.860992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.860974    4214 scope.go:122] "RemoveContainer" containerID="1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"
Oct 30 03:04:51.861354 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:04:51.861326    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659\": container with ID starting with 1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659 not found: ID does not exist" containerID="1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"
Oct 30 03:04:51.861453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.861365    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659"} err="failed to get container status \"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659\": rpc error: code = NotFound desc = could not find container \"1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659\": container with ID starting with 1d626aae352cea8a244408da181812cc6ddeaa96c2d9b81964baddfbed7cb659 not found: ID does not exist"
Oct 30 03:04:51.876528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.876475    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 03:04:51.883322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.883297    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-ngdvc"]
Oct 30 03:04:51.920035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.919694    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 03:04:51.927245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.927213    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-d4z2s"]
Oct 30 03:04:51.963265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.963224    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:51.965048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.964988    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 03:04:51.971648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:51.971625    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9gx7c"]
Oct 30 03:04:52.005944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:52.005913    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 03:04:52.012633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:52.012608    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-7cnf4"]
Oct 30 03:04:52.064746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:52.064549    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:04:52.615417 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:04:52.615378    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625bfcdd_107b_496c_bda0_f64657b9e2cf.slice/crio-355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff WatchSource:0}: Error finding container 355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff: Status 404 returned error can't find the container with id 355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff
Oct 30 03:04:52.700491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:52.687666    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:04:52.717465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:52.713422    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:04:53.074485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.074453    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d30799-48f2-4909-841b-cb2f91fffd0e" path="/var/lib/kubelet/pods/09d30799-48f2-4909-841b-cb2f91fffd0e/volumes"
Oct 30 03:04:53.075273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.075248    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432ca094-9b1c-4d92-9223-91f93b9bb441" path="/var/lib/kubelet/pods/432ca094-9b1c-4d92-9223-91f93b9bb441/volumes"
Oct 30 03:04:53.076152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.076130    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e774eaf-0d2a-462f-ae16-ad0e86be3496" path="/var/lib/kubelet/pods/5e774eaf-0d2a-462f-ae16-ad0e86be3496/volumes"
Oct 30 03:04:53.077098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.077076    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1d97f4-1c98-4fce-951b-ac3de38bf081" path="/var/lib/kubelet/pods/cb1d97f4-1c98-4fce-951b-ac3de38bf081/volumes"
Oct 30 03:04:53.077495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.077477    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:04:53.077597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.077535    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" event={"ID":"625bfcdd-107b-496c-bda0-f64657b9e2cf","Type":"ContainerStarted","Data":"355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff"}
Oct 30 03:04:53.077597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.077569    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" event={"ID":"2aafd8ff-a536-42ec-9c86-446d2c05a767","Type":"ContainerStarted","Data":"68a7a93470183e4be1e9d9b42bd2deebcd7ad8585c11afba6ac69a7e446a33e3"}
Oct 30 03:04:53.118311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:53.118217    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" podStartSLOduration=16.118200067 podStartE2EDuration="16.118200067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:53.116556595 +0000 UTC m=+1432.190727392" watchObservedRunningTime="2026-10-30 03:04:53.118200067 +0000 UTC m=+1432.192370860"
Oct 30 03:04:54.088291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:54.088241    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" event={"ID":"625bfcdd-107b-496c-bda0-f64657b9e2cf","Type":"ContainerStarted","Data":"2e477c2e28c76a333ebb2b05756177f3bdced4cb75b2c7bfcd3bb01293183db5"}
Oct 30 03:04:54.088780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:54.088326    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:04:54.090037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:54.090005    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" event={"ID":"2aafd8ff-a536-42ec-9c86-446d2c05a767","Type":"ContainerStarted","Data":"f045cb45e039b64138864ad4ad8b2cf537fe5a4ee74d3ae3c39e6ac0a3a1ee81"}
Oct 30 03:04:54.132585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:54.132531    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" podStartSLOduration=8.753552904 podStartE2EDuration="9.132514519s" totalImagesPullingTime="378.961615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:54.129829379 +0000 UTC m=+1433.204000184" watchObservedRunningTime="2026-10-30 03:04:54.132514519 +0000 UTC m=+1433.206685313"
Oct 30 03:04:58.029830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:58.029791    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"]
Oct 30 03:04:58.042832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:58.042802    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5886317-27gpv"]
Oct 30 03:04:59.069587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:04:59.069552    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fcdafb-a368-4f00-815e-ba3be4d39723" path="/var/lib/kubelet/pods/a8fcdafb-a368-4f00-815e-ba3be4d39723/volumes"
Oct 30 03:05:00.045923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.045895    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:05:00.050571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.050551    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.073005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.072975    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:05:00.074905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.074882    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d30e9f-serving-cert\""
Oct 30 03:05:00.121309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.121225    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.121818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.121786    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.122056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.122030    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.122479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.122452    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkvr\" (UniqueName: \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.145148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.145120    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8809-b1c465"]
Oct 30 03:05:00.151198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.151179    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:00.162514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.162487    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-b1c465"]
Oct 30 03:05:00.167973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.167948    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 03:05:00.223735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.223693    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlk6r\" (UniqueName: \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\") pod \"test-curl-8809-b1c465\" (UID: \"43c24187-c95b-48ea-ba1b-da4c23b8582c\") " pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:00.223917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.223769    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkvr\" (UniqueName: \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.224053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.224027    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.224107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.224087    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.224256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.224219    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.224983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.224958    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.226588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.226568    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.226932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.226907    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.244246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.244206    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkvr\" (UniqueName: \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\") pod \"maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.331367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.330475    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlk6r\" (UniqueName: \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\") pod \"test-curl-8809-b1c465\" (UID: \"43c24187-c95b-48ea-ba1b-da4c23b8582c\") " pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:00.364669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.356002    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlk6r\" (UniqueName: \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\") pod \"test-curl-8809-b1c465\" (UID: \"43c24187-c95b-48ea-ba1b-da4c23b8582c\") " pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:00.370823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.370547    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:00.474948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.474899    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:00.844132 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:00.844103    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50970dbb_e48e_4275_8bee_78c0d97bd665.slice/crio-d698ef5fc75cb7b3e341fd3650e0669e56f0570a9c1593296281e9f35c66f8d1 WatchSource:0}: Error finding container d698ef5fc75cb7b3e341fd3650e0669e56f0570a9c1593296281e9f35c66f8d1: Status 404 returned error can't find the container with id d698ef5fc75cb7b3e341fd3650e0669e56f0570a9c1593296281e9f35c66f8d1
Oct 30 03:05:00.844404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.844386    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:05:00.868372 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:00.868325    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43c24187_c95b_48ea_ba1b_da4c23b8582c.slice/crio-28202c72b86f78043acf96ad122d76c3bd4d0353d6b7f0b38befb2eff90183e1 WatchSource:0}: Error finding container 28202c72b86f78043acf96ad122d76c3bd4d0353d6b7f0b38befb2eff90183e1: Status 404 returned error can't find the container with id 28202c72b86f78043acf96ad122d76c3bd4d0353d6b7f0b38befb2eff90183e1
Oct 30 03:05:00.868852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:00.868831    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-b1c465"]
Oct 30 03:05:01.114574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:01.114504    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:05:01.199510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:01.199481    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" event={"ID":"50970dbb-e48e-4275-8bee-78c0d97bd665","Type":"ContainerStarted","Data":"d698ef5fc75cb7b3e341fd3650e0669e56f0570a9c1593296281e9f35c66f8d1"}
Oct 30 03:05:01.202126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:01.202096    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-b1c465" event={"ID":"43c24187-c95b-48ea-ba1b-da4c23b8582c","Type":"ContainerStarted","Data":"533a5487d2214a84357f344d4d161fa929a882252c9981fff5ffa732718b9890"}
Oct 30 03:05:01.202356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:01.202337    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-b1c465" event={"ID":"43c24187-c95b-48ea-ba1b-da4c23b8582c","Type":"ContainerStarted","Data":"28202c72b86f78043acf96ad122d76c3bd4d0353d6b7f0b38befb2eff90183e1"}
Oct 30 03:05:01.246065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:01.246007    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8809-b1c465" podStartSLOduration=1.24599278 podStartE2EDuration="1.24599278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:01.240539473 +0000 UTC m=+1440.314710266" watchObservedRunningTime="2026-10-30 03:05:01.24599278 +0000 UTC m=+1440.320163572"
Oct 30 03:05:02.213215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:02.213172    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" event={"ID":"50970dbb-e48e-4275-8bee-78c0d97bd665","Type":"ContainerStarted","Data":"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"}
Oct 30 03:05:02.213705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:02.213281    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:02.256280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:02.250361    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8809-b1c465"]
Oct 30 03:05:02.256846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:02.256825    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8809-b1c465"]
Oct 30 03:05:02.279114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:02.279058    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" podStartSLOduration=1.8622815529999999 podStartE2EDuration="2.279038698s" totalImagesPullingTime="416.757145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:02.276331135 +0000 UTC m=+1441.350501942" watchObservedRunningTime="2026-10-30 03:05:02.279038698 +0000 UTC m=+1441.353209495"
Oct 30 03:05:03.223605 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.223556    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8809-b1c465" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" containerName="test-curl-8809-b1c465" containerID="cri-o://533a5487d2214a84357f344d4d161fa929a882252c9981fff5ffa732718b9890" gracePeriod=2
Oct 30 03:05:03.538188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.538108    4214 scope.go:122] "RemoveContainer" containerID="a6b97f194376faeff78086408720046ce932c689c30c80a2950e2c01089da230"
Oct 30 03:05:03.567011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.566404    4214 scope.go:122] "RemoveContainer" containerID="8f2883131b0fb34c140e1e6d93711fe8e4de2ed86a629d8b140f8579506b5d84"
Oct 30 03:05:03.567011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.566826    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:05:03.568144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.568123    4214 cpu_manager.go:414] "containerMap: removing container" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" containerName="test-curl-8809-b1c465"
Oct 30 03:05:03.568144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.568144    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" containerName="test-curl-8809-b1c465"
Oct 30 03:05:03.568439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.568423    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" containerName="test-curl-8809-b1c465"
Oct 30 03:05:03.572964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.572943    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.591955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.591912    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a3337a-openshift-default-dockercfg-f96w5\""
Oct 30 03:05:03.594635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.594613    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:05:03.596005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.595983    4214 scope.go:122] "RemoveContainer" containerID="9c3547a96e7dc84f3be52ec599c46214e39519cbcaebf3a86a20733837aad017"
Oct 30 03:05:03.607936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.607917    4214 scope.go:122] "RemoveContainer" containerID="e04096487540cc12695d5eaaa1aeee82acea0f94937ff395ad2d77553e242724"
Oct 30 03:05:03.620405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.620378    4214 scope.go:122] "RemoveContainer" containerID="26abb4738da54ecae30d485693e8140720eb6f46f181449a2ee7a06e6d5b19d3"
Oct 30 03:05:03.629562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629380    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.629562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629471    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhl99\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.629761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629570    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.629761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629605    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.629761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629661    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.629761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629754    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.630004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629866    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.630004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629926    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.630004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.629953    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.632099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.632071    4214 scope.go:122] "RemoveContainer" containerID="4f832b6729c2bc0ee4a9b3683cfb4da67d4ae407dad8f0d661c1bb6dc6614f6c"
Oct 30 03:05:03.646026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.645891    4214 scope.go:122] "RemoveContainer" containerID="278c87c4328eefb0af54b8529f49b7545dcf5dea592dbedde9a965e4cf36f646"
Oct 30 03:05:03.670786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.667562    4214 scope.go:122] "RemoveContainer" containerID="555a256a193b4371875a47acbdfa60c71f9a9a45d503e41a6efb410f303e1ea5"
Oct 30 03:05:03.753251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.752693    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.753251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.752801    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl99\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.766700    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.766791    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.766874    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.766963    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.767339    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.767470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.768255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.767524    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.803261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.802964    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.821252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.819554    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.821417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.821276    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.821781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.821752    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.862315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.842219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.862315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.857876    4214 scope.go:122] "RemoveContainer" containerID="62d6b75d908a41974a2a3e0fe1cceaf95464318eae922e395733cde0bde0471f"
Oct 30 03:05:03.881259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.866456    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl99\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.897323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.897289    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:03.951397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:03.945402    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:04.001404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.001370    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\") pod \"e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") " pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:04.204244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.204207    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:04.822538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.822501    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:05:04.823023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.822716    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:04.823216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.823191    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:04.823401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:04.823379    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:05.030570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.030536    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"]
Oct 30 03:05:05.036938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.036813    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7347e427-5vm8f"]
Oct 30 03:05:05.071688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.071601    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994fd1b9-4e15-49a3-9446-5630966e7517" path="/var/lib/kubelet/pods/994fd1b9-4e15-49a3-9446-5630966e7517/volumes"
Oct 30 03:05:05.103022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.102991    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:05:05.391140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.391090    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerStarted","Data":"462cd71628422504745e1ac26b3ceb22037dc5970698c42314c7d874cf994495"}
Oct 30 03:05:05.391350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.391171    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerStarted","Data":"9cb1c25f0f6d998c86db5cc99ddd78565edf50208e854795f57ae7d68dd7c73d"}
Oct 30 03:05:05.394033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.394003    4214 generic.go:356] "Generic (PLEG): container finished" podID="43c24187-c95b-48ea-ba1b-da4c23b8582c" containerID="533a5487d2214a84357f344d4d161fa929a882252c9981fff5ffa732718b9890" exitCode=137
Oct 30 03:05:05.442437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.442380    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podStartSLOduration=2.442360682 podStartE2EDuration="2.442360682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:05.434804753 +0000 UTC m=+1444.508975547" watchObservedRunningTime="2026-10-30 03:05:05.442360682 +0000 UTC m=+1444.516531476"
Oct 30 03:05:05.507385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.507360    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:05.634345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.632784    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\" (UniqueName: \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\") pod \"43c24187-c95b-48ea-ba1b-da4c23b8582c\" (UID: \"43c24187-c95b-48ea-ba1b-da4c23b8582c\") "
Oct 30 03:05:05.651543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.651482    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r" pod "43c24187-c95b-48ea-ba1b-da4c23b8582c" (UID: "43c24187-c95b-48ea-ba1b-da4c23b8582c"). InnerVolumeSpecName "kube-api-access-jlk6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:05.735612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:05.735571    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlk6r\" (UniqueName: \"kubernetes.io/projected/43c24187-c95b-48ea-ba1b-da4c23b8582c-kube-api-access-jlk6r\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:06.205168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.205132    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:06.212257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.212214    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:06.524306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.522125    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:06.525324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.525302    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:06.525513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.525500    4214 scope.go:122] "RemoveContainer" containerID="533a5487d2214a84357f344d4d161fa929a882252c9981fff5ffa732718b9890"
Oct 30 03:05:06.553256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:06.551517    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:05:06.681502 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:06.681456    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8809-b1c465\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" pod="opendatahub/test-curl-8809-b1c465"
Oct 30 03:05:07.073509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:07.073476    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c24187-c95b-48ea-ba1b-da4c23b8582c" path="/var/lib/kubelet/pods/43c24187-c95b-48ea-ba1b-da4c23b8582c/volumes"
Oct 30 03:05:08.229095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:08.229053    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:05:09.169281 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:09.169243    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:05:09.929585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:09.929549    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:05:09.935428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:09.935401    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:09.953765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:09.953739    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:05:09.972580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:09.972425    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a3337a-dockercfg-qq7jz\""
Oct 30 03:05:09.972580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:09.972488    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a3337a\""
Oct 30 03:05:10.052744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.052708    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp6nw\" (UniqueName: \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.052971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.052954    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.156264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.155917    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.156264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.156220    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp6nw\" (UniqueName: \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.157775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.157745    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.210306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.198007    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp6nw\" (UniqueName: \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\") pod \"payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:10.286252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:10.284146    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:11.017432 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:11.017388    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb40834a_d7e8_4f20_86ac_0d28a2f81664.slice/crio-29436c77ce10ab4349afda3d2051b96f6638f00cff8a672fe771660415e7d3b1 WatchSource:0}: Error finding container 29436c77ce10ab4349afda3d2051b96f6638f00cff8a672fe771660415e7d3b1: Status 404 returned error can't find the container with id 29436c77ce10ab4349afda3d2051b96f6638f00cff8a672fe771660415e7d3b1
Oct 30 03:05:11.017959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:11.017937    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:05:11.701506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:11.701467    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" event={"ID":"cb40834a-d7e8-4f20-86ac-0d28a2f81664","Type":"ContainerStarted","Data":"2f16efc484070e0fa141e1c7780f3ec566c9dd731c5f96a30f8a39949d8c9061"}
Oct 30 03:05:11.701744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:11.701715    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" event={"ID":"cb40834a-d7e8-4f20-86ac-0d28a2f81664","Type":"ContainerStarted","Data":"29436c77ce10ab4349afda3d2051b96f6638f00cff8a672fe771660415e7d3b1"}
Oct 30 03:05:11.701850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:11.701756    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:11.747176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:11.747127    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" podStartSLOduration=2.747111741 podStartE2EDuration="2.747111741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:11.740919029 +0000 UTC m=+1450.815089823" watchObservedRunningTime="2026-10-30 03:05:11.747111741 +0000 UTC m=+1450.821282534"
Oct 30 03:05:15.775887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.775843    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 03:05:15.776735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.776164    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" containerID="cri-o://7b3906d400e780300fc60047165a8ff20e915dc32d78736803d0027402695836" gracePeriod=30
Oct 30 03:05:15.789281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.789245    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 03:05:15.789654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.789514    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy" containerID="cri-o://305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56" gracePeriod=30
Oct 30 03:05:15.799439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.799393    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 03:05:15.799686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.799657    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy" containerID="cri-o://cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed" gracePeriod=30
Oct 30 03:05:15.805886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.805860    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 03:05:15.806109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.806062    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" containerID="cri-o://6bf4c15f985337f54589d9307d3569535151f56096c9406592c7441c5373c855" gracePeriod=30
Oct 30 03:05:15.816531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.816499    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 03:05:15.816719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.816698    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy" containerID="cri-o://cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed" gracePeriod=30
Oct 30 03:05:15.822168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.822143    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 03:05:15.822566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.822524    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy" containerID="cri-o://504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450" gracePeriod=30
Oct 30 03:05:15.831618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.831582    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 03:05:15.832138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.832076    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy" containerID="cri-o://9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d" gracePeriod=30
Oct 30 03:05:15.839476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.839454    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 03:05:15.839620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:15.839602    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" containerID="cri-o://35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747" gracePeriod=30
Oct 30 03:05:16.598201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.598163    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 03:05:16.598480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.598455    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy" containerID="cri-o://1e1ba056b95940286ada873f2228db76cb824a4b0cefac8c65afac82b2979c8a" gracePeriod=30
Oct 30 03:05:16.604078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.604051    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 03:05:16.604308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.604280    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy" containerID="cri-o://91b3ea2e98bed1bdd8a28b8f566b0fec5751e9bb03ef27478b2a426bc85c45b9" gracePeriod=30
Oct 30 03:05:16.610289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.610221    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 03:05:16.610594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.610469    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" containerID="cri-o://83e8fbbdfc96cb8499bc3f44b3b16875519ed1dbb5b873da931bee68e6d41823" gracePeriod=30
Oct 30 03:05:16.616448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.616424    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 03:05:16.616657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:16.616625    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy" containerID="cri-o://04567e9b203076f5156733e697b6f0ea950aadeea120d92a53f8f347f1eb623c" gracePeriod=30
Oct 30 03:05:17.325577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:17.325543    4214 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:05:17.326118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:17.325613    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:05:17.791908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:17.791820    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:05:17.791908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:17.791896    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:05:18.139412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:18.139370    4214 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:05:18.139695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:18.139659    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:05:20.746253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:20.745448    4214 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:05:20.746253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:20.745536    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:05:21.425858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.424593    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 03:05:21.478520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.476971    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.478579    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.478644    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.478751    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.478853    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.478944    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.479035    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.479118    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.479353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.479208    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\") pod \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\" (UID: \"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e\") "
Oct 30 03:05:21.481469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.481408    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.482023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.481815    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.482023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.481896    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.482023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.481929    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:21.482023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.481999    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.482800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.482775    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "kube-api-access-dhskm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.484793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.484774    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:21.484908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.484783    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.485944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.485914    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token" pod "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" (UID: "5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.524827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.524756    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.535082    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.536714    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538037    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538215    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538293    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538360    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538371    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538385    4214 cpu_manager.go:414] "containerMap: removing container" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538393    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538704    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538720    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538749    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538760    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.538774    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.545941    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:05:21.547960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.546037    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.582345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.582311    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.582745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.582724    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.582927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.582910    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.583118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583103    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.583241 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583211    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.583441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583422    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.583561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583548    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.583662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583651    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.583787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583772    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.583920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.583907    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.584057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584042    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.584183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584170    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.584332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584317    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.584470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584457    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.584572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584558    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.584702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584690    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.584843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.584803    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.585020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585005    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\") pod \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\" (UID: \"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0\") "
Oct 30 03:05:21.585142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585128    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.585304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585288    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.585493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585478    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.585615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585598    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.585712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585699    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.585801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585789    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.585944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.585931    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.586064 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.586050    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\") pod \"d52d195d-da08-4ec1-a536-67d421d3f369\" (UID: \"d52d195d-da08-4ec1-a536-67d421d3f369\") "
Oct 30 03:05:21.586172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.586159    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\") pod \"79f816da-469c-44b8-8431-73198aa7ecde\" (UID: \"79f816da-469c-44b8-8431-73198aa7ecde\") "
Oct 30 03:05:21.586656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.586627    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rwc\" (UniqueName: \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.587467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.587442    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.587741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.587723    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.587844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.587834    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.587919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.587909    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.587984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.587974    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588043    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhskm\" (UniqueName: \"kubernetes.io/projected/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-kube-api-access-dhskm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588111    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588244    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588347    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588435    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.588793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.588772    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.590593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.590565    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.592109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.591866    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.592751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.592492    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.600728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.600417    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.601279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601102    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.602061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601438    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:21.602061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601627    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.602061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601697    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:21.602061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601805    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.602061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.601975    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.603027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.602666    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.603027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.602597    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:21.603027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.602869    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.603553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.603313    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:21.604116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.604087    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.614266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.611807    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "kube-api-access-2gdj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.615620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.614420    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.615620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.614420    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "kube-api-access-44k6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.615620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.614480    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.615934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.615898    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:21.618556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.618520    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.618797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.618764    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "kube-api-access-h4jfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.622752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.622721    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.654297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.653547    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo" pod "79f816da-469c-44b8-8431-73198aa7ecde" (UID: "79f816da-469c-44b8-8431-73198aa7ecde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:21.654546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.654505    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy" pod "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" (UID: "a9d038a8-3f1b-48c1-8b13-dcedb72e74b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:21.659495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.659455    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token" pod "d52d195d-da08-4ec1-a536-67d421d3f369" (UID: "d52d195d-da08-4ec1-a536-67d421d3f369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:21.686511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.680502    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.687391    4214 cpu_manager.go:414] "containerMap: removing container" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.687415    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.687468    4214 cpu_manager.go:414] "containerMap: removing container" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.687475    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f816da-469c-44b8-8431-73198aa7ecde" containerName="istio-proxy"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.695876    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.695996    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rwc\" (UniqueName: \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696280    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdj4\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-kube-api-access-2gdj4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696297    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52d195d-da08-4ec1-a536-67d421d3f369-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696307    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696316    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696324    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696332    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696341    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696349    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696360    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696373    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696384    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696392    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696401    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696409    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f816da-469c-44b8-8431-73198aa7ecde-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696419    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696426    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696437    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52d195d-da08-4ec1-a536-67d421d3f369-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696450    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44k6f\" (UniqueName: \"kubernetes.io/projected/79f816da-469c-44b8-8431-73198aa7ecde-kube-api-access-44k6f\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696459    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52d195d-da08-4ec1-a536-67d421d3f369-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696468    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f816da-469c-44b8-8431-73198aa7ecde-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696476    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52d195d-da08-4ec1-a536-67d421d3f369-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696485    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f816da-469c-44b8-8431-73198aa7ecde-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696492    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696500    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jfv\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-kube-api-access-h4jfv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696508    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696520    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.696530    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:21.706983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.697741    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.720396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.719640    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.724248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.724177    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:05:21.734711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.734673    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c1abc2-serving-cert\""
Oct 30 03:05:21.780797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.780384    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rwc\" (UniqueName: \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\") pod \"payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.808028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.807966    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.821546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.820487    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4f4p\" (UniqueName: \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.821546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.820671    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.821546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.820744    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.870661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.870383    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:21.924785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.922370    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.924785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.922451    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.924785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.922707    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.924785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.922799    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4f4p\" (UniqueName: \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.924785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.924057    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.928245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.928204    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:21.953322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:21.953291    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:22.000698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.000632    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4f4p\" (UniqueName: \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\") pod \"maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:22.068949 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.068749    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:22.069116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.068969    4214 generic.go:356] "Generic (PLEG): container finished" podID="733d454d-eeae-4735-9665-6d42e983e61f" containerID="7b3906d400e780300fc60047165a8ff20e915dc32d78736803d0027402695836" exitCode=0
Oct 30 03:05:22.069116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.069067    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" event={"ID":"733d454d-eeae-4735-9665-6d42e983e61f","Type":"ContainerDied","Data":"7b3906d400e780300fc60047165a8ff20e915dc32d78736803d0027402695836"}
Oct 30 03:05:22.069116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.069113    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f" event={"ID":"733d454d-eeae-4735-9665-6d42e983e61f","Type":"ContainerDied","Data":"102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5"}
Oct 30 03:05:22.069313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.069135    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102273d8f488f3527a16a91a75ac48cac787f19e5c1db89f6b8699a41cf0d9f5"
Oct 30 03:05:22.075384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.075032    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 03:05:22.079567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.079548    4214 generic.go:356] "Generic (PLEG): container finished" podID="e79b0562-ee2d-4598-9769-f50e121aa764" containerID="91b3ea2e98bed1bdd8a28b8f566b0fec5751e9bb03ef27478b2a426bc85c45b9" exitCode=0
Oct 30 03:05:22.080067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.079669    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" event={"ID":"e79b0562-ee2d-4598-9769-f50e121aa764","Type":"ContainerDied","Data":"91b3ea2e98bed1bdd8a28b8f566b0fec5751e9bb03ef27478b2a426bc85c45b9"}
Oct 30 03:05:22.081988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.081787    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 03:05:22.082302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.082264    4214 generic.go:356] "Generic (PLEG): container finished" podID="c2ce4243-031c-4132-a5e8-946854676d44" containerID="6bf4c15f985337f54589d9307d3569535151f56096c9406592c7441c5373c855" exitCode=0
Oct 30 03:05:22.082532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.082396    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" event={"ID":"c2ce4243-031c-4132-a5e8-946854676d44","Type":"ContainerDied","Data":"6bf4c15f985337f54589d9307d3569535151f56096c9406592c7441c5373c855"}
Oct 30 03:05:22.082611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.082596    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md" event={"ID":"c2ce4243-031c-4132-a5e8-946854676d44","Type":"ContainerDied","Data":"ed8881dbeb43f44dc4d1161410e901ed8069a427a438de26f808d2bf9d5b9461"}
Oct 30 03:05:22.082668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.082623    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed8881dbeb43f44dc4d1161410e901ed8069a427a438de26f808d2bf9d5b9461"
Oct 30 03:05:22.084276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.084256    4214 generic.go:356] "Generic (PLEG): container finished" podID="79f816da-469c-44b8-8431-73198aa7ecde" containerID="35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747" exitCode=0
Oct 30 03:05:22.084387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.084339    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerDied","Data":"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"}
Oct 30 03:05:22.084387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.084378    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp" event={"ID":"79f816da-469c-44b8-8431-73198aa7ecde","Type":"ContainerDied","Data":"ac67e4fe412dec98136cbb265ae70d9ff8b3015c792c548572f57f0361e762d7"}
Oct 30 03:05:22.084606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.084408    4214 scope.go:122] "RemoveContainer" containerID="35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"
Oct 30 03:05:22.084699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.084671    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"
Oct 30 03:05:22.088215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.088151    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 03:05:22.114111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.113410    4214 generic.go:356] "Generic (PLEG): container finished" podID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerID="1e1ba056b95940286ada873f2228db76cb824a4b0cefac8c65afac82b2979c8a" exitCode=0
Oct 30 03:05:22.114111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.113592    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" event={"ID":"bbce61e7-9bf4-4504-b488-ca489556d1cf","Type":"ContainerDied","Data":"1e1ba056b95940286ada873f2228db76cb824a4b0cefac8c65afac82b2979c8a"}
Oct 30 03:05:22.140300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.140166    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 03:05:22.143606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.143548    4214 generic.go:356] "Generic (PLEG): container finished" podID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" containerID="504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450" exitCode=0
Oct 30 03:05:22.143914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.143860    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"
Oct 30 03:05:22.145858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.145335    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" event={"ID":"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e","Type":"ContainerDied","Data":"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"}
Oct 30 03:05:22.145858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.145385    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl" event={"ID":"5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e","Type":"ContainerDied","Data":"854836d0f5043475b952e0a2c934e55a33f3f5cd00e818b0fc071d3ff8492b63"}
Oct 30 03:05:22.148475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148409    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.148475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.148871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.148871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148759    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.148871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148807    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.149252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.148849    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.149252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149087    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.149252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149128    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.149252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149156    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.149764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149524    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.149764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149630    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.149944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.149751    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.150103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150025    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.150281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150091    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.150281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150213    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.150618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150263    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.150618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150517    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.150618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150566    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\") pod \"c2ce4243-031c-4132-a5e8-946854676d44\" (UID: \"c2ce4243-031c-4132-a5e8-946854676d44\") "
Oct 30 03:05:22.150960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150809    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.150960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150860    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.150960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.150911    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.151498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.151412    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.151498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.151461    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\") pod \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\" (UID: \"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60\") "
Oct 30 03:05:22.152958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.151803    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.152958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.151839    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.152958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.152822    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.152958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.152882    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\") pod \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\" (UID: \"e49c6ffe-a90e-4cfb-9322-22abdaedea6c\") "
Oct 30 03:05:22.162924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.161843    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.162924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.161977    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 03:05:22.162924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.162215    4214 generic.go:356] "Generic (PLEG): container finished" podID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerID="83e8fbbdfc96cb8499bc3f44b3b16875519ed1dbb5b873da931bee68e6d41823" exitCode=0
Oct 30 03:05:22.162924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.162345    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" event={"ID":"3ea50e0f-8890-4576-b68b-27a43b7073b9","Type":"ContainerDied","Data":"83e8fbbdfc96cb8499bc3f44b3b16875519ed1dbb5b873da931bee68e6d41823"}
Oct 30 03:05:22.170563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.169597    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.173758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.173716    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.176204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.173909    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.176204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.174580    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.177433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.176808    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.177433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177069    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.177433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177151    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.177433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177308    4214 generic.go:356] "Generic (PLEG): container finished" podID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerID="9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d" exitCode=0
Oct 30 03:05:22.177675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177457    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" event={"ID":"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60","Type":"ContainerDied","Data":"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"}
Oct 30 03:05:22.177675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177509    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn" event={"ID":"bb72f1b6-3856-4efe-8ad1-e22d0cf22b60","Type":"ContainerDied","Data":"d10b530b1e265cf71b7c25344bf31aa0a5614846248ad39db153d1c46a2f2f0f"}
Oct 30 03:05:22.177675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.177595    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"
Oct 30 03:05:22.179489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.178490    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.179703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.179677    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.179846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.179668    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.180016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.179994    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.181205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.181166    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.181945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.181919    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.182636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.182606    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.182853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.182827    4214 generic.go:356] "Generic (PLEG): container finished" podID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerID="04567e9b203076f5156733e697b6f0ea950aadeea120d92a53f8f347f1eb623c" exitCode=0
Oct 30 03:05:22.187224 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.187200    4214 scope.go:122] "RemoveContainer" containerID="068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"
Oct 30 03:05:22.188930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.188906    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" event={"ID":"aee1419b-2461-4811-8e23-2fd1905bdb0e","Type":"ContainerDied","Data":"04567e9b203076f5156733e697b6f0ea950aadeea120d92a53f8f347f1eb623c"}
Oct 30 03:05:22.191416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.191383    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.197827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.197799    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 03:05:22.206539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.204910    4214 generic.go:356] "Generic (PLEG): container finished" podID="d52d195d-da08-4ec1-a536-67d421d3f369" containerID="cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed" exitCode=0
Oct 30 03:05:22.206539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.205015    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" event={"ID":"d52d195d-da08-4ec1-a536-67d421d3f369","Type":"ContainerDied","Data":"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"}
Oct 30 03:05:22.206539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.205058    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j" event={"ID":"d52d195d-da08-4ec1-a536-67d421d3f369","Type":"ContainerDied","Data":"a9a4b08415ffa6d1d294116b35754cdc646f28699d64dbbc86b9563b52fbc345"}
Oct 30 03:05:22.206539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.205140    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"
Oct 30 03:05:22.211448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.211419    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.214069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.212716    4214 scope.go:122] "RemoveContainer" containerID="35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"
Oct 30 03:05:22.214069 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.213399    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747\": container with ID starting with 35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747 not found: ID does not exist" containerID="35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"
Oct 30 03:05:22.214069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.213433    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747"} err="failed to get container status \"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747\": rpc error: code = NotFound desc = could not find container \"35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747\": container with ID starting with 35889b250b80614a999a3f04d9bfc4e571e8e99dd6a0e4ec016dcba97d4ca747 not found: ID does not exist"
Oct 30 03:05:22.214069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.213458    4214 scope.go:122] "RemoveContainer" containerID="068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"
Oct 30 03:05:22.214837 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.214501    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31\": container with ID starting with 068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31 not found: ID does not exist" containerID="068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"
Oct 30 03:05:22.214837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.214536    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31"} err="failed to get container status \"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31\": rpc error: code = NotFound desc = could not find container \"068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31\": container with ID starting with 068d957a8ef5e44286e51a751c8965b27a18356f98eaef4354ea4862f9766c31 not found: ID does not exist"
Oct 30 03:05:22.214837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.214557    4214 scope.go:122] "RemoveContainer" containerID="504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"
Oct 30 03:05:22.215024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.214871    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "kube-api-access-pm575". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.215081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215028    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.215304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215272    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token" pod "e49c6ffe-a90e-4cfb-9322-22abdaedea6c" (UID: "e49c6ffe-a90e-4cfb-9322-22abdaedea6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.215734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215706    4214 generic.go:356] "Generic (PLEG): container finished" podID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerID="cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed" exitCode=0
Oct 30 03:05:22.215813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215799    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" event={"ID":"e49c6ffe-a90e-4cfb-9322-22abdaedea6c","Type":"ContainerDied","Data":"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"}
Oct 30 03:05:22.215869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215842    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv" event={"ID":"e49c6ffe-a90e-4cfb-9322-22abdaedea6c","Type":"ContainerDied","Data":"86168fb4f5e8f84193d1fcc8fb9503313a0f67918989a2bf4c446b58f0ef76c0"}
Oct 30 03:05:22.215941 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.215923    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"
Oct 30 03:05:22.219142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.218107    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.219142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.218763    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.222017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.221658    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.222017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.221869    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.222945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.222334    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.222945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.222825    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v" pod "c2ce4243-031c-4132-a5e8-946854676d44" (UID: "c2ce4243-031c-4132-a5e8-946854676d44"). InnerVolumeSpecName "kube-api-access-2bm8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.223116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223097    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc" pod "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" (UID: "bb72f1b6-3856-4efe-8ad1-e22d0cf22b60"). InnerVolumeSpecName "kube-api-access-ndxvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.223341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223301    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-kptwp"]
Oct 30 03:05:22.223923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223894    4214 generic.go:356] "Generic (PLEG): container finished" podID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" containerID="305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56" exitCode=0
Oct 30 03:05:22.224114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223938    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" event={"ID":"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0","Type":"ContainerDied","Data":"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"}
Oct 30 03:05:22.224114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223945    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"
Oct 30 03:05:22.224114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.223975    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p" event={"ID":"a9d038a8-3f1b-48c1-8b13-dcedb72e74b0","Type":"ContainerDied","Data":"57d0a83d9513d6582310849edcd0fd3a70e1d00ee95f096d1cf1d622d2641884"}
Oct 30 03:05:22.274320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.261679    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.271535    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280129    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.275028    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "kube-api-access-fwxmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280168    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280213    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280326    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280357    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280417    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280443    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280499    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280564    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280587    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280621    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280704    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280735    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280846    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.280951    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.281097    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\") pod \"3ea50e0f-8890-4576-b68b-27a43b7073b9\" (UID: \"3ea50e0f-8890-4576-b68b-27a43b7073b9\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.281106    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.281139    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\") pod \"733d454d-eeae-4735-9665-6d42e983e61f\" (UID: \"733d454d-eeae-4735-9665-6d42e983e61f\") "
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.281825    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.281986    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282010    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm575\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-kube-api-access-pm575\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282026    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282040    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282054    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282066    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282081    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282094    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282108    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bm8v\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-kube-api-access-2bm8v\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282123    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282137    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ce4243-031c-4132-a5e8-946854676d44-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282136    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282150    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282164    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxmv\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-kube-api-access-fwxmv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282177    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ce4243-031c-4132-a5e8-946854676d44-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282191    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282203    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282216    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ce4243-031c-4132-a5e8-946854676d44-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282246    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea50e0f-8890-4576-b68b-27a43b7073b9-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282294    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282307    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282320    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282333    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282348    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282363    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282380    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ce4243-031c-4132-a5e8-946854676d44-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282394    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndxvc\" (UniqueName: \"kubernetes.io/projected/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-kube-api-access-ndxvc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282408    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282421    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282435    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49c6ffe-a90e-4cfb-9322-22abdaedea6c-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282468    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.282681    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.283024    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.283056    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.283325    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.283363    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.283699    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.289757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.284321    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.296841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.290928    4214 scope.go:122] "RemoveContainer" containerID="504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"
Oct 30 03:05:22.296841 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.291423    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450\": container with ID starting with 504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450 not found: ID does not exist" containerID="504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"
Oct 30 03:05:22.296841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.291458    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450"} err="failed to get container status \"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450\": rpc error: code = NotFound desc = could not find container \"504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450\": container with ID starting with 504b18e1e209c0c47260e85ef5db1b8580fcd653588f636eae440425a89a6450 not found: ID does not exist"
Oct 30 03:05:22.296841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.291482    4214 scope.go:122] "RemoveContainer" containerID="83e8fbbdfc96cb8499bc3f44b3b16875519ed1dbb5b873da931bee68e6d41823"
Oct 30 03:05:22.301514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.301492    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.315371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.308175    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.315371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.314672    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.323327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.320714    4214 scope.go:122] "RemoveContainer" containerID="9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"
Oct 30 03:05:22.323327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.320918    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.323327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.320981    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.323327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.321046    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978" pod "3ea50e0f-8890-4576-b68b-27a43b7073b9" (UID: "3ea50e0f-8890-4576-b68b-27a43b7073b9"). InnerVolumeSpecName "kube-api-access-sc978". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.323327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.321100    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy" pod "733d454d-eeae-4735-9665-6d42e983e61f" (UID: "733d454d-eeae-4735-9665-6d42e983e61f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.385133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.385106    4214 scope.go:122] "RemoveContainer" containerID="9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"
Oct 30 03:05:22.385586 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.385562    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d\": container with ID starting with 9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d not found: ID does not exist" containerID="9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"
Oct 30 03:05:22.385669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.385596    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d"} err="failed to get container status \"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d\": rpc error: code = NotFound desc = could not find container \"9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d\": container with ID starting with 9b051592ef6c82c24414f1043b520887cb620d31414e063abffea298765e784d not found: ID does not exist"
Oct 30 03:05:22.385669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.385614    4214 scope.go:122] "RemoveContainer" containerID="cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"
Oct 30 03:05:22.392417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392386    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392492    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392607    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392679    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392782    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392844    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.392934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.392875    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.393146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.393124    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\") pod \"e79b0562-ee2d-4598-9769-f50e121aa764\" (UID: \"e79b0562-ee2d-4598-9769-f50e121aa764\") "
Oct 30 03:05:22.393910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.393884    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733d454d-eeae-4735-9665-6d42e983e61f-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.393915    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733d454d-eeae-4735-9665-6d42e983e61f-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.393930    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc978\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-kube-api-access-sc978\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.393989    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394018    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394031    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394044    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394057    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733d454d-eeae-4735-9665-6d42e983e61f-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394093    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394108    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394122    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394141    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394154    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733d454d-eeae-4735-9665-6d42e983e61f-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394165    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.394173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394180    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea50e0f-8890-4576-b68b-27a43b7073b9-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.395479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.394441    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.395479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.395056    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.401057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.396706    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.401057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.396896    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.401057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.397320    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.403416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.403286    4214 scope.go:122] "RemoveContainer" containerID="cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"
Oct 30 03:05:22.403957 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.403790    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed\": container with ID starting with cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed not found: ID does not exist" containerID="cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"
Oct 30 03:05:22.403957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.403827    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed"} err="failed to get container status \"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed\": rpc error: code = NotFound desc = could not find container \"cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed\": container with ID starting with cf631b877abd9b697900e20147654d80185b6446126cfc2f020a8fa60eb4baed not found: ID does not exist"
Oct 30 03:05:22.403957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.403854    4214 scope.go:122] "RemoveContainer" containerID="cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"
Oct 30 03:05:22.406560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.406524    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.417570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.413485    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 03:05:22.417570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.416449    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "kube-api-access-bbns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.417570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.416933    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.418035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.417997    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token" pod "e79b0562-ee2d-4598-9769-f50e121aa764" (UID: "e79b0562-ee2d-4598-9769-f50e121aa764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.427717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.427635    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 03:05:22.427885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.427857    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-9hsjl"]
Oct 30 03:05:22.430704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.430684    4214 scope.go:122] "RemoveContainer" containerID="cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"
Oct 30 03:05:22.430995 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.430976    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed\": container with ID starting with cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed not found: ID does not exist" containerID="cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"
Oct 30 03:05:22.431092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.431013    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed"} err="failed to get container status \"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed\": rpc error: code = NotFound desc = could not find container \"cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed\": container with ID starting with cb3df80a61108ee530e7c1cb750c4a8003495c3ea2eee8db4614ab15f28d80ed not found: ID does not exist"
Oct 30 03:05:22.431092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.431035    4214 scope.go:122] "RemoveContainer" containerID="305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"
Oct 30 03:05:22.445791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.445677    4214 scope.go:122] "RemoveContainer" containerID="305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"
Oct 30 03:05:22.446049 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:22.446023    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56\": container with ID starting with 305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56 not found: ID does not exist" containerID="305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"
Oct 30 03:05:22.446141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.446058    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56"} err="failed to get container status \"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56\": rpc error: code = NotFound desc = could not find container \"305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56\": container with ID starting with 305c0d26a0eea92a58af89d13d512d2e8773fedb2b21776724cb10327c6f8a56 not found: ID does not exist"
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513312    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513430    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513477    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513520    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513618    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513697    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513721    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513754    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.513921    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\") pod \"bbce61e7-9bf4-4504-b488-ca489556d1cf\" (UID: \"bbce61e7-9bf4-4504-b488-ca489556d1cf\") "
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514537    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514700    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbns7\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-kube-api-access-bbns7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514724    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79b0562-ee2d-4598-9769-f50e121aa764-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514738    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514740    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514753    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79b0562-ee2d-4598-9769-f50e121aa764-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514767    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514780    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514796    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514810    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514825    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79b0562-ee2d-4598-9769-f50e121aa764-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514837    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79b0562-ee2d-4598-9769-f50e121aa764-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514989    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.514994    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.517320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.515254    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.518813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.517791    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.520070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.519193    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.532271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.530895    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "kube-api-access-l75gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.532797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.532766    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token" pod "bbce61e7-9bf4-4504-b488-ca489556d1cf" (UID: "bbce61e7-9bf4-4504-b488-ca489556d1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.557713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.557221    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 03:05:22.597790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.585774    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 03:05:22.603344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.602589    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-gjw7j"]
Oct 30 03:05:22.614208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.614166    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:05:22.619775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.619750    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.619905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.619819    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.619905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.619858    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.619948    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620009    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620163    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620199    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620292    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620346    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\") pod \"aee1419b-2461-4811-8e23-2fd1905bdb0e\" (UID: \"aee1419b-2461-4811-8e23-2fd1905bdb0e\") "
Oct 30 03:05:22.620514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620479    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.620632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620606    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:22.620894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620872    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.621015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.620983    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee1419b-2461-4811-8e23-2fd1905bdb0e-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621016    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621030    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbce61e7-9bf4-4504-b488-ca489556d1cf-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621043    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621056    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621069    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621082    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621094    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621106    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbce61e7-9bf4-4504-b488-ca489556d1cf-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621119    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75gf\" (UniqueName: \"kubernetes.io/projected/bbce61e7-9bf4-4504-b488-ca489556d1cf-kube-api-access-l75gf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.621426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.621241    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.622431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.622407    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.623598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.623564    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.623696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.623593    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:05:22.625167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.625140    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:22.625703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.625668    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q" pod "aee1419b-2461-4811-8e23-2fd1905bdb0e" (UID: "aee1419b-2461-4811-8e23-2fd1905bdb0e"). InnerVolumeSpecName "kube-api-access-tdf7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:22.648991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.648936    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 03:05:22.654297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.654271    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-rrr2p"]
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723592    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723628    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723644    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723655    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723667    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdf7q\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-kube-api-access-tdf7q\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723682    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee1419b-2461-4811-8e23-2fd1905bdb0e-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.725829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.723695    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee1419b-2461-4811-8e23-2fd1905bdb0e-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:05:22.730579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.730553    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:05:22.744426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.744333    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 03:05:22.755167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.754930    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:05:22.768128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.768081    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-w5qzv"]
Oct 30 03:05:22.774708 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:22.774661    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b4399a_07e2_45f1_a2c0_308b2765a636.slice/crio-6b040d48eed4ac37b45433578a13d42e6d4a0f279bc614ac291936a43e8cda79 WatchSource:0}: Error finding container 6b040d48eed4ac37b45433578a13d42e6d4a0f279bc614ac291936a43e8cda79: Status 404 returned error can't find the container with id 6b040d48eed4ac37b45433578a13d42e6d4a0f279bc614ac291936a43e8cda79
Oct 30 03:05:22.981550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.981509    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 03:05:22.989330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:22.989281    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-7wxgn"]
Oct 30 03:05:23.076886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.076858    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e" path="/var/lib/kubelet/pods/5f6c48f2-cdb6-47fe-9ae9-97cdb4ae544e/volumes"
Oct 30 03:05:23.077690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.077669    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f816da-469c-44b8-8431-73198aa7ecde" path="/var/lib/kubelet/pods/79f816da-469c-44b8-8431-73198aa7ecde/volumes"
Oct 30 03:05:23.078306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.078289    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d038a8-3f1b-48c1-8b13-dcedb72e74b0" path="/var/lib/kubelet/pods/a9d038a8-3f1b-48c1-8b13-dcedb72e74b0/volumes"
Oct 30 03:05:23.079210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.079194    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" path="/var/lib/kubelet/pods/bb72f1b6-3856-4efe-8ad1-e22d0cf22b60/volumes"
Oct 30 03:05:23.079818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.079802    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52d195d-da08-4ec1-a536-67d421d3f369" path="/var/lib/kubelet/pods/d52d195d-da08-4ec1-a536-67d421d3f369/volumes"
Oct 30 03:05:23.080501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.080483    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" path="/var/lib/kubelet/pods/e49c6ffe-a90e-4cfb-9322-22abdaedea6c/volumes"
Oct 30 03:05:23.236820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.236797    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"
Oct 30 03:05:23.236954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.236794    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2" event={"ID":"aee1419b-2461-4811-8e23-2fd1905bdb0e","Type":"ContainerDied","Data":"770d55591ff4237d55f018e0297620f8367d8e5b3babe6b3f14541bd564b1fc8"}
Oct 30 03:05:23.236954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.236916    4214 scope.go:122] "RemoveContainer" containerID="04567e9b203076f5156733e697b6f0ea950aadeea120d92a53f8f347f1eb623c"
Oct 30 03:05:23.243380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.243357    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"
Oct 30 03:05:23.243700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.243672    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22" event={"ID":"e79b0562-ee2d-4598-9769-f50e121aa764","Type":"ContainerDied","Data":"df42a5c1332b82b1fb70f71d809f347dc20df380832b1188d8565433d3a89227"}
Oct 30 03:05:23.245129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.245107    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" event={"ID":"c5b4399a-07e2-45f1-a2c0-308b2765a636","Type":"ContainerStarted","Data":"6b040d48eed4ac37b45433578a13d42e6d4a0f279bc614ac291936a43e8cda79"}
Oct 30 03:05:23.247263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.247207    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" event={"ID":"57908fae-37b2-4542-9d96-aa109aba4bb0","Type":"ContainerStarted","Data":"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"}
Oct 30 03:05:23.247366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.247330    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" event={"ID":"57908fae-37b2-4542-9d96-aa109aba4bb0","Type":"ContainerStarted","Data":"709d3856279f9afca9790edb2b0a5068298d3a7f141038b63fb244628431a6db"}
Oct 30 03:05:23.247431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.247376    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:23.250171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.250151    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"
Oct 30 03:05:23.250171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.250160    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q" event={"ID":"bbce61e7-9bf4-4504-b488-ca489556d1cf","Type":"ContainerDied","Data":"b14b25e5f0ad441c94bad5982321013f122d920b9b1d627eb044a8f4897e9530"}
Oct 30 03:05:23.251744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.251726    4214 scope.go:122] "RemoveContainer" containerID="91b3ea2e98bed1bdd8a28b8f566b0fec5751e9bb03ef27478b2a426bc85c45b9"
Oct 30 03:05:23.254573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.254492    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"
Oct 30 03:05:23.254573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.254498    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j" event={"ID":"3ea50e0f-8890-4576-b68b-27a43b7073b9","Type":"ContainerDied","Data":"69df8969e17124c9d68514d14b7ae2b66dd4dd1e0539bc678b08d587404a13c7"}
Oct 30 03:05:23.254909 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.254882    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"
Oct 30 03:05:23.255120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.254891    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"
Oct 30 03:05:23.276822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.276797    4214 scope.go:122] "RemoveContainer" containerID="1e1ba056b95940286ada873f2228db76cb824a4b0cefac8c65afac82b2979c8a"
Oct 30 03:05:23.311026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.310994    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 03:05:23.315787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.315765    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-pnzg2"]
Oct 30 03:05:23.353295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.353152    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 03:05:23.359841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.359814    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-djd22"]
Oct 30 03:05:23.399535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.399489    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" podStartSLOduration=14.399475941 podStartE2EDuration="14.399475941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:23.394621386 +0000 UTC m=+1462.468792217" watchObservedRunningTime="2026-10-30 03:05:23.399475941 +0000 UTC m=+1462.473646734"
Oct 30 03:05:23.438779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.438749    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 03:05:23.449332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.449306    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jcf5f"]
Oct 30 03:05:23.490979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.490948    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 03:05:23.498288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.498264    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-8gd6q"]
Oct 30 03:05:23.628770 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.628688    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 03:05:23.644539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.644499    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-xm26j"]
Oct 30 03:05:23.674949 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.674914    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 03:05:23.682643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:23.682614    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-jl4md"]
Oct 30 03:05:24.028342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.028271    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:05:24.029071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029056    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy"
Oct 30 03:05:24.029120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029073    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy"
Oct 30 03:05:24.029120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029092    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy"
Oct 30 03:05:24.029120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029097    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy"
Oct 30 03:05:24.029120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029112    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy"
Oct 30 03:05:24.029120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029118    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029133    4214 cpu_manager.go:414] "containerMap: removing container" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029138    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029147    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029152    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029167    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029171    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029181    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029186    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029191    4214 cpu_manager.go:414] "containerMap: removing container" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy"
Oct 30 03:05:24.029292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029196    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029323    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029334    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029342    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029349    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="733d454d-eeae-4735-9665-6d42e983e61f" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029359    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb72f1b6-3856-4efe-8ad1-e22d0cf22b60" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029366    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49c6ffe-a90e-4cfb-9322-22abdaedea6c" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029373    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ce4243-031c-4132-a5e8-946854676d44" containerName="istio-proxy"
Oct 30 03:05:24.029577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.029381    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" containerName="istio-proxy"
Oct 30 03:05:24.032281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.032261    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.058738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.058708    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.058889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.058796    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.058889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.058820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.058994    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tb7\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.059077    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.059168    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.059223    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.059526    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.059659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.059589    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.061154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.061129    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:05:24.160772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.160737    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.160947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.160814    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.160947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.160868    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161042    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161106    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161150    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161180    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161207    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tb7\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.161477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161419    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.162419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161517    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.162419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161205    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.162419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.161666    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.162419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.162326    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.164167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.164136    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.165163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.165138    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.183166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.183134    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.183363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.183342    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tb7\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.271645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.271608    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" event={"ID":"c5b4399a-07e2-45f1-a2c0-308b2765a636","Type":"ContainerStarted","Data":"ea57aaf3b9ee751b6e60f6ced996e63f5839ca1ef50b3ec9582950e6a1915da0"}
Oct 30 03:05:24.271851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.271683    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:24.319051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.318994    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" podStartSLOduration=9.871952432 podStartE2EDuration="10.318981612s" totalImagesPullingTime="447.02918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:24.314950001 +0000 UTC m=+1463.389120794" watchObservedRunningTime="2026-10-30 03:05:24.318981612 +0000 UTC m=+1463.393152405"
Oct 30 03:05:24.347989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.347958    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:24.695741 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:24.695702    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d19a2_81ff_4e61_8e21_cdb01f814b39.slice/crio-0a3679dcc4d126aee897938fad13e32c6396b936a7aab0775bf7f4b535ccdb44 WatchSource:0}: Error finding container 0a3679dcc4d126aee897938fad13e32c6396b936a7aab0775bf7f4b535ccdb44: Status 404 returned error can't find the container with id 0a3679dcc4d126aee897938fad13e32c6396b936a7aab0775bf7f4b535ccdb44
Oct 30 03:05:24.696321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.696295    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:05:24.703470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.703440    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:24.703541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.703504    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:24.703541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:24.703532    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:25.075905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.075870    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea50e0f-8890-4576-b68b-27a43b7073b9" path="/var/lib/kubelet/pods/3ea50e0f-8890-4576-b68b-27a43b7073b9/volumes"
Oct 30 03:05:25.076687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.076668    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733d454d-eeae-4735-9665-6d42e983e61f" path="/var/lib/kubelet/pods/733d454d-eeae-4735-9665-6d42e983e61f/volumes"
Oct 30 03:05:25.077322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.077302    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee1419b-2461-4811-8e23-2fd1905bdb0e" path="/var/lib/kubelet/pods/aee1419b-2461-4811-8e23-2fd1905bdb0e/volumes"
Oct 30 03:05:25.078393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.078375    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbce61e7-9bf4-4504-b488-ca489556d1cf" path="/var/lib/kubelet/pods/bbce61e7-9bf4-4504-b488-ca489556d1cf/volumes"
Oct 30 03:05:25.079004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.078987    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ce4243-031c-4132-a5e8-946854676d44" path="/var/lib/kubelet/pods/c2ce4243-031c-4132-a5e8-946854676d44/volumes"
Oct 30 03:05:25.079644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.079627    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79b0562-ee2d-4598-9769-f50e121aa764" path="/var/lib/kubelet/pods/e79b0562-ee2d-4598-9769-f50e121aa764/volumes"
Oct 30 03:05:25.286760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.286720    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" event={"ID":"2e5d19a2-81ff-4e61-8e21-cdb01f814b39","Type":"ContainerStarted","Data":"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"}
Oct 30 03:05:25.286959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.286776    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" event={"ID":"2e5d19a2-81ff-4e61-8e21-cdb01f814b39","Type":"ContainerStarted","Data":"0a3679dcc4d126aee897938fad13e32c6396b936a7aab0775bf7f4b535ccdb44"}
Oct 30 03:05:25.342850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.342751    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podStartSLOduration=2.342734045 podStartE2EDuration="2.342734045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:25.341962914 +0000 UTC m=+1464.416133708" watchObservedRunningTime="2026-10-30 03:05:25.342734045 +0000 UTC m=+1464.416904838"
Oct 30 03:05:25.349140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.349116    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:25.351357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.351322    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:25.351458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:25.351373    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:26.360383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:26.360345    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:26.360995 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:26.360962    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:27.356252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:27.354631    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:27.356252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:27.354692    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:28.350623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:28.350584    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:28.351200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:28.351167    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:29.353648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:29.353612    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:29.354202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:29.354166    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:30.292152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:30.292120    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:05:30.353783 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:30.353747    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:30.354400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:30.354364    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:31.349161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:31.349115    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 03:05:31.349383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:31.349182    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 03:05:32.319425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.318776    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:05:32.331830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.331780    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:05:32.332016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.331892    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.412254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.410421    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:32.412254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.412090    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.412705    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ab09ae-openshift-default-dockercfg-qgtq4\""
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414402    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414400    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414478    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414563    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414593    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414636    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414799    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414842    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.414911    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.415005    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58f6\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516571    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516610    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516740    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516874    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516937    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.516998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.516984    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517154    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517195    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517315    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58f6\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517379    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517421    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517481    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.517886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.517864    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.518210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.518191    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.519428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.519405    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.519779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.519758    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.557171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.550921    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.557171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.553953    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58f6\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\") pod \"e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") " pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:32.672369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:32.659389    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:33.089057 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:33.089026    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f636458_995c_42ef_a64e_a2fd496443fa.slice/crio-34786626bf45c04f06c365bd87de5bbdb70d4858fc5de5bcd190665b5cebba89 WatchSource:0}: Error finding container 34786626bf45c04f06c365bd87de5bbdb70d4858fc5de5bcd190665b5cebba89: Status 404 returned error can't find the container with id 34786626bf45c04f06c365bd87de5bbdb70d4858fc5de5bcd190665b5cebba89
Oct 30 03:05:33.089193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.089158    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:05:33.091967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.091935    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:33.092048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.092002    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:33.092048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.092033    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:33.464832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.464733    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerStarted","Data":"a566b47bd0b7e0d3c4f1853293de38d76de54c412280401fbaa29be100fbb4ee"}
Oct 30 03:05:33.464832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.464801    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerStarted","Data":"34786626bf45c04f06c365bd87de5bbdb70d4858fc5de5bcd190665b5cebba89"}
Oct 30 03:05:33.529691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.529627    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podStartSLOduration=2.529608094 podStartE2EDuration="2.529608094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:33.524618836 +0000 UTC m=+1472.598789628" watchObservedRunningTime="2026-10-30 03:05:33.529608094 +0000 UTC m=+1472.603778889"
Oct 30 03:05:33.663044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:33.663000    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:34.278146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:34.278113    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:05:34.670151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:34.670118    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:35.487306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:35.487270    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:35.488431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:35.488407    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:05:37.942141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:37.942103    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:05:37.948257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:37.948210    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:37.964723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:37.964692    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:05:37.976964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:37.976933    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ab09ae-dockercfg-hrkwb\""
Oct 30 03:05:37.977130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:37.977039    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ab09ae\""
Oct 30 03:05:38.168262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.167621    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.168262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.167718    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2sf\" (UniqueName: \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.255298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.246276    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:05:38.255298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.251315    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.298258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.274273    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.304257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.300374    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2sf\" (UniqueName: \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.340261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.307435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.364262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.332399    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:05:38.435948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.408317    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqs7\" (UniqueName: \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.435948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.415988    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2sf\" (UniqueName: \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\") pod \"payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.435948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.432906    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.535124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.535033    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqs7\" (UniqueName: \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.535314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.535188    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.536335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.536308    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.561686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.561652    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqs7\" (UniqueName: \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\") pod \"payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.592167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.592140    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:38.753019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.752987    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:38.930161 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:38.930130    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c29d85_597b_45f2_a692_b8d8d5c9d9d5.slice/crio-bb66f3787d253f8d69c88c4b818f8727c5c3dc9801f0e6aba8b1f1ea9a9ee7e3 WatchSource:0}: Error finding container bb66f3787d253f8d69c88c4b818f8727c5c3dc9801f0e6aba8b1f1ea9a9ee7e3: Status 404 returned error can't find the container with id bb66f3787d253f8d69c88c4b818f8727c5c3dc9801f0e6aba8b1f1ea9a9ee7e3
Oct 30 03:05:38.930788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:38.930750    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:05:39.099376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.099341    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:05:39.099736 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:39.099344    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12702691_000a_49da_8c40_9464c13b953c.slice/crio-f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85 WatchSource:0}: Error finding container f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85: Status 404 returned error can't find the container with id f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85
Oct 30 03:05:39.170177 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:05:39.170144    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:05:39.527380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.527282    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" event={"ID":"80c29d85-597b-45f2-a692-b8d8d5c9d9d5","Type":"ContainerStarted","Data":"049701df9920ac16df24fe02fa7730478a5074090f8197192558b40e23692b0f"}
Oct 30 03:05:39.527380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.527351    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" event={"ID":"80c29d85-597b-45f2-a692-b8d8d5c9d9d5","Type":"ContainerStarted","Data":"bb66f3787d253f8d69c88c4b818f8727c5c3dc9801f0e6aba8b1f1ea9a9ee7e3"}
Oct 30 03:05:39.527625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.527408    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:39.529347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.529312    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" event={"ID":"12702691-000a-49da-8c40-9464c13b953c","Type":"ContainerStarted","Data":"c77bdf3a96e8f8cd1828eb3fbaef98ef30b6c95ff455a6d74ef18d31205e523d"}
Oct 30 03:05:39.529488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.529358    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" event={"ID":"12702691-000a-49da-8c40-9464c13b953c","Type":"ContainerStarted","Data":"f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85"}
Oct 30 03:05:39.529488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.529386    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:39.576699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.576646    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" podStartSLOduration=2.576631464 podStartE2EDuration="2.576631464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:39.573654403 +0000 UTC m=+1478.647825208" watchObservedRunningTime="2026-10-30 03:05:39.576631464 +0000 UTC m=+1478.650802259"
Oct 30 03:05:39.619523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:39.619477    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" podStartSLOduration=1.619464086 podStartE2EDuration="1.619464086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:39.616505686 +0000 UTC m=+1478.690676480" watchObservedRunningTime="2026-10-30 03:05:39.619464086 +0000 UTC m=+1478.693634878"
Oct 30 03:05:43.557305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:43.553373    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.709269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:43.700893    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.866254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:43.852912    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.978495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:43.978443    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.079265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.076719    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.124209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.124117    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.269276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.258337    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:05:44.269276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.258405    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.679310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.663615    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.682794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.682763    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:05:44.722828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.704348    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.743504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:44.743472    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:05:45.088432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.051718    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.088432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.062356    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:05:45.117121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.104947    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.141292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.130570    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:05:45.310251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.307302    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.685148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.684896    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.730253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.727321    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:45.816493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.816458    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:05:45.816778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.816759    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:05:45.816926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.816914    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:05:45.817073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.817061    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:05:45.817193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.817181    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:05:45.832297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.832270    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:05:45.833051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:45.833032    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:05:48.873816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:48.873783    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:05:48.878565 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:48.878532    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:48.900184 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:48.900160    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:05:48.901441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:48.901276    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a3337a-serving-cert\""
Oct 30 03:05:49.003629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.003586    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.003813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.003691    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.003951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.003926    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdvg\" (UniqueName: \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.004126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.004108    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.109439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.106935    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdvg\" (UniqueName: \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.109439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.107161    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.109439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.107216    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.109439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.107278    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.109439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.108359    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.119534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.119461    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.128309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.120668    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.164038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.163966    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdvg\" (UniqueName: \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\") pod \"maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.203949 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.203610    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:49.589176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.589144    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:05:49.590400 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:49.590372    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a20fad_af87_46e2_a6bc_f2c09fc60872.slice/crio-e754fc65c7335c964f7a9124d5218ef8cd8921e6fbc4b21091463dc878dacf77 WatchSource:0}: Error finding container e754fc65c7335c964f7a9124d5218ef8cd8921e6fbc4b21091463dc878dacf77: Status 404 returned error can't find the container with id e754fc65c7335c964f7a9124d5218ef8cd8921e6fbc4b21091463dc878dacf77
Oct 30 03:05:49.662246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.652466    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:05:49.662246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.659924    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.688278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.686970    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cea203-openshift-default-dockercfg-t8d85\""
Oct 30 03:05:49.688278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.687756    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:05:49.728774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.728733    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729077    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729220    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpswq\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729380    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729601 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729707    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.729918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.729902    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.730048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.730033    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.730166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.730152    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.834981    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835051    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpswq\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835099    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835218    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835268    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835383    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835435    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835472    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.835626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.835610    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.840292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.836144    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.851251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.845186    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.851251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.845463    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.851251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.845692    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.851251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.847146    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.916279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.900642    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:49.918277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:49.918245    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:50.046268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.030778    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:50.046268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.039286    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpswq\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:50.118292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.118186    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" event={"ID":"d8a20fad-af87-46e2-a6bc-f2c09fc60872","Type":"ContainerStarted","Data":"e754fc65c7335c964f7a9124d5218ef8cd8921e6fbc4b21091463dc878dacf77"}
Oct 30 03:05:50.311252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.310968    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:50.665307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.654554    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:05:50.670827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.669791    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:05:50.827998 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:05:50.827962    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4b7242_82ab_4a88_b113_cbc054c7f4cd.slice/crio-063bd6a69f8a05b32b162f4e11877d538cce863322d1b8b6b8987fb5ea5e1ab1 WatchSource:0}: Error finding container 063bd6a69f8a05b32b162f4e11877d538cce863322d1b8b6b8987fb5ea5e1ab1: Status 404 returned error can't find the container with id 063bd6a69f8a05b32b162f4e11877d538cce863322d1b8b6b8987fb5ea5e1ab1
Oct 30 03:05:50.829526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.829496    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:05:50.849751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.834417    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:50.849751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.834507    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:50.849751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:50.834550    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:51.205434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.205185    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" event={"ID":"d8a20fad-af87-46e2-a6bc-f2c09fc60872","Type":"ContainerStarted","Data":"3c317f68c6fb4c9326890418ac38a56a8e1306708b329d10efb3dc7ecd5682b4"}
Oct 30 03:05:51.205891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.205468    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:51.208432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.208401    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerStarted","Data":"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"}
Oct 30 03:05:51.208554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.208456    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerStarted","Data":"063bd6a69f8a05b32b162f4e11877d538cce863322d1b8b6b8987fb5ea5e1ab1"}
Oct 30 03:05:51.249731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.249683    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" podStartSLOduration=2.66754831 podStartE2EDuration="3.249668945s" totalImagesPullingTime="582.120635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:51.248677903 +0000 UTC m=+1490.322848703" watchObservedRunningTime="2026-10-30 03:05:51.249668945 +0000 UTC m=+1490.323839738"
Oct 30 03:05:51.306289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.303629    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podStartSLOduration=2.303611371 podStartE2EDuration="2.303611371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:51.29420564 +0000 UTC m=+1490.368376436" watchObservedRunningTime="2026-10-30 03:05:51.303611371 +0000 UTC m=+1490.377782242"
Oct 30 03:05:51.312356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.312301    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:51.313797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.313754    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:05:51.313932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:51.313811    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:05:52.316924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:52.316895    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:53.229518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.229488    4214 generic.go:356] "Generic (PLEG): container finished" podID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerID="cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab" exitCode=0
Oct 30 03:05:53.229735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.229559    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerDied","Data":"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"}
Oct 30 03:05:53.230049 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.230028    4214 scope.go:122] "RemoveContainer" containerID="cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"
Oct 30 03:05:53.230119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.230048    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:53.231123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.231102    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:05:53.231916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.231887    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:53.231980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.231968    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:53.232025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:53.232002    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:05:54.268082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:54.267978    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerStarted","Data":"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"}
Oct 30 03:05:54.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:54.982340    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:05:54.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:54.982681    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 03:05:54.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:54.982730    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 03:05:55.544770 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.544562    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:05:55.818455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.818376    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:05:55.822953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.822935    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:55.879589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.879548    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cea203\""
Oct 30 03:05:55.880314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.880030    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cea203-dockercfg-h2fw6\""
Oct 30 03:05:55.891330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.891292    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:55.891492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.891354    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:55.900328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.900302    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:05:56.079370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:55.998054    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:56.097415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.097372    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:56.099337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.099308    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:56.099530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.037587    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 03:05:56.099692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.099667    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 03:05:56.432696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.432613    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:05:56.445479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.445447    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.488974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.488688    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:05:56.491710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.490321    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") pod \"payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:56.567326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.542420    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:56.628993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.628953    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5thn\" (UniqueName: \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.629573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.629546    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.732399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.731884    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5thn\" (UniqueName: \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.732399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.731963    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.733222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.732656    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.782237 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.782188    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5thn\" (UniqueName: \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\") pod \"payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:56.925571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.925541    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:05:56.979850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.978500    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 03:05:56.979850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:56.978561    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 03:05:57.075766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.075509    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:57.232242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.230278    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:05:57.498507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.498469    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" event={"ID":"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5","Type":"ContainerStarted","Data":"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"}
Oct 30 03:05:57.498671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.498533    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" event={"ID":"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5","Type":"ContainerStarted","Data":"e9d3063d66665545abb014d06df3aa43dbe23c63234d3ea5f762202ce22cea62"}
Oct 30 03:05:57.498671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.498557    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:05:57.582059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.582036    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:05:57.583854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.583801    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" podStartSLOduration=2.5837839540000003 podStartE2EDuration="2.583783954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:57.569467945 +0000 UTC m=+1496.643638739" watchObservedRunningTime="2026-10-30 03:05:57.583783954 +0000 UTC m=+1496.657954748"
Oct 30 03:05:57.955254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.948796    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 03:05:57.955254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:57.948855    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 03:05:58.509336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.509296    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" event={"ID":"9577a1a5-4200-41e9-a6ec-5274f09990aa","Type":"ContainerStarted","Data":"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"}
Oct 30 03:05:58.509515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.509351    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" event={"ID":"9577a1a5-4200-41e9-a6ec-5274f09990aa","Type":"ContainerStarted","Data":"1b5c1c90003b5af863584575b1557e5b44082145b3dd3f4c13e1b5258684648e"}
Oct 30 03:05:58.509572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.509551    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:05:58.554250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.551619    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" podStartSLOduration=3.551600637 podStartE2EDuration="3.551600637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:58.550566508 +0000 UTC m=+1497.624737300" watchObservedRunningTime="2026-10-30 03:05:58.551600637 +0000 UTC m=+1497.625771430"
Oct 30 03:05:58.957346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.957312    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 03:05:58.957932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:05:58.957899    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 03:06:00.042374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:00.042293    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:06:00.044860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:00.044830    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:06:01.027323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.027289    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 03:06:01.027550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.027524    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy" containerID="cri-o://8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896" gracePeriod=30
Oct 30 03:06:01.046584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.046550    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:06:01.047496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.047455    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy" containerID="cri-o://2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022" gracePeriod=30
Oct 30 03:06:01.074162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.067721    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 03:06:01.074162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.067969    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy" containerID="cri-o://de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2" gracePeriod=30
Oct 30 03:06:01.075743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.074831    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 03:06:01.075743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:01.075050    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy" containerID="cri-o://44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24" gracePeriod=30
Oct 30 03:06:02.036308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:02.036275    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"]
Oct 30 03:06:02.053187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:02.053155    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-071e1cb2-rrqjf"]
Oct 30 03:06:03.069896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:03.069859    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b04868a-85a5-4035-824f-7c4bef768e74" path="/var/lib/kubelet/pods/6b04868a-85a5-4035-824f-7c4bef768e74/volumes"
Oct 30 03:06:04.507989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.507957    4214 scope.go:122] "RemoveContainer" containerID="162f143a183059202dff2061e2c1b0801a169348a4dd410089e16eb915c2c3ac"
Oct 30 03:06:04.519312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.519291    4214 scope.go:122] "RemoveContainer" containerID="68cf81f186bc8b951d8f6d126e3a7c8b47a77da0475d22a2b44463272db4be51"
Oct 30 03:06:04.529175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.529152    4214 scope.go:122] "RemoveContainer" containerID="7d5acac63389ea6bbac4d7fd14e2cd1a8cfeddabc105f39de888fdc53613bc3c"
Oct 30 03:06:04.539637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.539615    4214 scope.go:122] "RemoveContainer" containerID="6bf4c15f985337f54589d9307d3569535151f56096c9406592c7441c5373c855"
Oct 30 03:06:04.549785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.549751    4214 scope.go:122] "RemoveContainer" containerID="082ae025149388a267ddd5c179e93b327839bb8c7e8ea34a5aae57eb178bb86e"
Oct 30 03:06:04.559923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.559903    4214 scope.go:122] "RemoveContainer" containerID="1b01618b36636e2e4686f8985022f165ae4eee93b5c19cb10c1263c929463a3d"
Oct 30 03:06:04.569840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.569819    4214 scope.go:122] "RemoveContainer" containerID="7b3906d400e780300fc60047165a8ff20e915dc32d78736803d0027402695836"
Oct 30 03:06:04.579791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.579774    4214 scope.go:122] "RemoveContainer" containerID="cff8032e3fa7e99cdaf6d831d71afa737f0bb9aee852ddba70d3a22d5e5e4391"
Oct 30 03:06:04.592948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:04.592929    4214 scope.go:122] "RemoveContainer" containerID="5bd36502db3ecc308f4b2d4312773a8d9959c2ce1536a1a8896d3bd0ba676a8a"
Oct 30 03:06:06.332575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.331991    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.473946    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474045    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474087    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474143    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474200    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474278    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474333    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474369    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.477693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.474418    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\") pod \"d74ec34e-e120-4de0-b38b-3f396ed51414\" (UID: \"d74ec34e-e120-4de0-b38b-3f396ed51414\") "
Oct 30 03:06:06.485880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.485843    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.486494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.486452    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.486494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.486467    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:06.486691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.486668    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.486853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.486837    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.509642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.509605    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "kube-api-access-sth7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.510032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.510003    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:06.510762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.510723    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.533573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.533521    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy" pod "d74ec34e-e120-4de0-b38b-3f396ed51414" (UID: "d74ec34e-e120-4de0-b38b-3f396ed51414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.578362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.578329    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588145    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588175    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588191    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588205    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74ec34e-e120-4de0-b38b-3f396ed51414-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588218    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588268    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sth7v\" (UniqueName: \"kubernetes.io/projected/d74ec34e-e120-4de0-b38b-3f396ed51414-kube-api-access-sth7v\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588284    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588298    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.588688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.588311    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74ec34e-e120-4de0-b38b-3f396ed51414-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.611157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.611129    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691378    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691462    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691517    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691544    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691593    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692350    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692392    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692438    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692629    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692693    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692748    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692797    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692826    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692869    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692895    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.691806    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.692033    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693019    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693111    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\") pod \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\" (UID: \"46021ffa-2f10-4e21-ac0c-cdf61d77b0af\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693171    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693199    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\") pod \"297a8a31-7402-4ebe-86df-bcd02ab9d096\" (UID: \"297a8a31-7402-4ebe-86df-bcd02ab9d096\") "
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693927    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297a8a31-7402-4ebe-86df-bcd02ab9d096-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693952    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.693966    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.694175    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.697412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.694515    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:06.698393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.698374    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.698709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.698691    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.699108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.698805    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.699108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.698911    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.700582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.700430    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.701402 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.701375    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.702856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.702835    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:06.702957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.702939    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "kube-api-access-8dfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.704796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.704620    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "kube-api-access-fl6hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.704906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.704874    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.704906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.704893    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:06.705087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.705068    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:06:06.706334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.706309    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy" pod "297a8a31-7402-4ebe-86df-bcd02ab9d096" (UID: "297a8a31-7402-4ebe-86df-bcd02ab9d096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.706444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.706393    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy" pod "46021ffa-2f10-4e21-ac0c-cdf61d77b0af" (UID: "46021ffa-2f10-4e21-ac0c-cdf61d77b0af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.794815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.794779    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.794972    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795041    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795099    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795123    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.795215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795137    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795212    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795287    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795343    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795420    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\") pod \"f82cfe94-cdc1-45a8-b793-f933637b8f39\" (UID: \"f82cfe94-cdc1-45a8-b793-f933637b8f39\") "
Oct 30 03:06:06.795486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795454    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.795700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795673    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.795883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.795864    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.796300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796278    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796306    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796320    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796331    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796344    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796357    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796372    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfkq\" (UniqueName: \"kubernetes.io/projected/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-kube-api-access-8dfkq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796385    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796398    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796410    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796423    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl6hx\" (UniqueName: \"kubernetes.io/projected/297a8a31-7402-4ebe-86df-bcd02ab9d096-kube-api-access-fl6hx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796437    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796451    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297a8a31-7402-4ebe-86df-bcd02ab9d096-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796464    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796478    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796508    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796523    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796537    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.796846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.796550    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46021ffa-2f10-4e21-ac0c-cdf61d77b0af-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.797376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.797190    4214 generic.go:356] "Generic (PLEG): container finished" podID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerID="44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24" exitCode=0
Oct 30 03:06:06.797376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.797361    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" event={"ID":"297a8a31-7402-4ebe-86df-bcd02ab9d096","Type":"ContainerDied","Data":"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"}
Oct 30 03:06:06.799281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.797550    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"
Oct 30 03:06:06.799281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.797839    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "kube-api-access-79bks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.799281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.798325    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p" event={"ID":"297a8a31-7402-4ebe-86df-bcd02ab9d096","Type":"ContainerDied","Data":"68f82328b8ba07955e43bccc052f3d8f05df0191b16fadc8f8abb5edbb9acb3d"}
Oct 30 03:06:06.799281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.798360    4214 scope.go:122] "RemoveContainer" containerID="44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"
Oct 30 03:06:06.799281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.798924    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:06.799614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.799383    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:06.799614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.799477    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:06.799848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.799824    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert" pod "f82cfe94-cdc1-45a8-b793-f933637b8f39" (UID: "f82cfe94-cdc1-45a8-b793-f933637b8f39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:06.804404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.804380    4214 generic.go:356] "Generic (PLEG): container finished" podID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerID="8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896" exitCode=0
Oct 30 03:06:06.804526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.804515    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" event={"ID":"d74ec34e-e120-4de0-b38b-3f396ed51414","Type":"ContainerDied","Data":"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"}
Oct 30 03:06:06.804592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.804548    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9" event={"ID":"d74ec34e-e120-4de0-b38b-3f396ed51414","Type":"ContainerDied","Data":"8ec4a33da3bd6e3ca8274d13aa4499cbbbda89a50c0d359f92660d6f4a57f2db"}
Oct 30 03:06:06.804656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.804622    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"
Oct 30 03:06:06.808599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.808015    4214 generic.go:356] "Generic (PLEG): container finished" podID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerID="2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022" exitCode=0
Oct 30 03:06:06.808599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.808108    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" event={"ID":"f82cfe94-cdc1-45a8-b793-f933637b8f39","Type":"ContainerDied","Data":"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"}
Oct 30 03:06:06.808599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.808143    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln" event={"ID":"f82cfe94-cdc1-45a8-b793-f933637b8f39","Type":"ContainerDied","Data":"b56747c469ceb9abab7f85d92918c283f48205676ba3b9531562b24115831f4b"}
Oct 30 03:06:06.808599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.808221    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"
Oct 30 03:06:06.811934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.811907    4214 generic.go:356] "Generic (PLEG): container finished" podID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerID="de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2" exitCode=0
Oct 30 03:06:06.812051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.812033    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" event={"ID":"46021ffa-2f10-4e21-ac0c-cdf61d77b0af","Type":"ContainerDied","Data":"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"}
Oct 30 03:06:06.812127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.812076    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw" event={"ID":"46021ffa-2f10-4e21-ac0c-cdf61d77b0af","Type":"ContainerDied","Data":"ec4cb666a7b39059e0fb17306c195b13857785f4f8965b82b745e0601a3e2e7a"}
Oct 30 03:06:06.812190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.812147    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"
Oct 30 03:06:06.814808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.814047    4214 scope.go:122] "RemoveContainer" containerID="44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"
Oct 30 03:06:06.814808 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:06.814696    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24\": container with ID starting with 44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24 not found: ID does not exist" containerID="44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"
Oct 30 03:06:06.814808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.814730    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24"} err="failed to get container status \"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24\": rpc error: code = NotFound desc = could not find container \"44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24\": container with ID starting with 44ee37756501fe2c27fc841eada1463db3cc99a96e0f5fefaa6467896adbbb24 not found: ID does not exist"
Oct 30 03:06:06.814808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.814752    4214 scope.go:122] "RemoveContainer" containerID="8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"
Oct 30 03:06:06.827726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.827695    4214 scope.go:122] "RemoveContainer" containerID="8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"
Oct 30 03:06:06.828068 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:06.828029    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896\": container with ID starting with 8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896 not found: ID does not exist" containerID="8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"
Oct 30 03:06:06.828149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.828075    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896"} err="failed to get container status \"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896\": rpc error: code = NotFound desc = could not find container \"8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896\": container with ID starting with 8a7c2c24b5610dcd191fe076ce83ba819dcf067cf755542168e4500bfbf6a896 not found: ID does not exist"
Oct 30 03:06:06.828149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.828091    4214 scope.go:122] "RemoveContainer" containerID="2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"
Oct 30 03:06:06.841076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.840991    4214 scope.go:122] "RemoveContainer" containerID="2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"
Oct 30 03:06:06.841355 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:06.841331    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022\": container with ID starting with 2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022 not found: ID does not exist" containerID="2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"
Oct 30 03:06:06.841438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.841365    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022"} err="failed to get container status \"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022\": rpc error: code = NotFound desc = could not find container \"2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022\": container with ID starting with 2d938661f36a49f16ba8c82a01d0cc17471f24197cb426e5a42a590804c52022 not found: ID does not exist"
Oct 30 03:06:06.841438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.841388    4214 scope.go:122] "RemoveContainer" containerID="de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"
Oct 30 03:06:06.852163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.852140    4214 scope.go:122] "RemoveContainer" containerID="de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"
Oct 30 03:06:06.852509 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:06.852490    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2\": container with ID starting with de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2 not found: ID does not exist" containerID="de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"
Oct 30 03:06:06.852579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.852515    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2"} err="failed to get container status \"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2\": rpc error: code = NotFound desc = could not find container \"de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2\": container with ID starting with de8dc63409a8d2d0ff453eca322b0e685051c29575a76c02f366aeeb457b09b2 not found: ID does not exist"
Oct 30 03:06:06.898580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.898546    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bks\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-kube-api-access-79bks\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.898580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.898575    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.898580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.898586    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.898886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.898594    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82cfe94-cdc1-45a8-b793-f933637b8f39-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.898886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.898603    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82cfe94-cdc1-45a8-b793-f933637b8f39-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:06.910443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.910413    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 03:06:06.915821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.915796    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-8nc5p"]
Oct 30 03:06:06.945817 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.945791    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 03:06:06.951095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.951074    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-lvtr9"]
Oct 30 03:06:06.985870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.985841    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:06:06.993176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:06.993146    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-rcwln"]
Oct 30 03:06:07.028655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.028631    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 03:06:07.033857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.033836    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-qhdhw"]
Oct 30 03:06:07.074892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.074829    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" path="/var/lib/kubelet/pods/297a8a31-7402-4ebe-86df-bcd02ab9d096/volumes"
Oct 30 03:06:07.075576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.075556    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" path="/var/lib/kubelet/pods/46021ffa-2f10-4e21-ac0c-cdf61d77b0af/volumes"
Oct 30 03:06:07.076263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.076216    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" path="/var/lib/kubelet/pods/d74ec34e-e120-4de0-b38b-3f396ed51414/volumes"
Oct 30 03:06:07.077389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:07.077368    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" path="/var/lib/kubelet/pods/f82cfe94-cdc1-45a8-b793-f933637b8f39/volumes"
Oct 30 03:06:08.512741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:08.512714    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:06:09.168387 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:09.168346    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:06:09.521946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:09.521860    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:06:16.695731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.695694    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 03:06:16.696356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.696009    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" containerID="cri-o://a339793b6d54cc6d352473a811285a1bbe0eefce70a0965086b54e3d764983b1" gracePeriod=30
Oct 30 03:06:16.707987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.707957    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 03:06:16.708191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.708172    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy" containerID="cri-o://8f61cbedeae0d3ea5051626651f80d71d7cf23f78d6206b4c1e1988e79c60b35" gracePeriod=30
Oct 30 03:06:16.712744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.712708    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 03:06:16.712876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.712859    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy" containerID="cri-o://5db97cb1864fa4951c0958cdff2a3462a2ba54c35f3baf25644e4ce42045fc6e" gracePeriod=30
Oct 30 03:06:16.720693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.720668    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 03:06:16.720871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:16.720841    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" containerID="cri-o://4c3c289cce4e6d30ea13825703128e3e3b6ec1405cf89d523bcc15ede0bc4a5c" gracePeriod=30
Oct 30 03:06:18.332576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.330649    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332672    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332694    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332710    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332723    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332771    4214 cpu_manager.go:414] "containerMap: removing container" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332778    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332811    4214 cpu_manager.go:414] "containerMap: removing container" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy"
Oct 30 03:06:18.333045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.332819    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy"
Oct 30 03:06:18.333474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.333132    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="46021ffa-2f10-4e21-ac0c-cdf61d77b0af" containerName="istio-proxy"
Oct 30 03:06:18.333474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.333161    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="297a8a31-7402-4ebe-86df-bcd02ab9d096" containerName="istio-proxy"
Oct 30 03:06:18.333474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.333198    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82cfe94-cdc1-45a8-b793-f933637b8f39" containerName="istio-proxy"
Oct 30 03:06:18.333474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.333207    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74ec34e-e120-4de0-b38b-3f396ed51414" containerName="istio-proxy"
Oct 30 03:06:18.352175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.352117    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.358330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.358297    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:06:18.375565 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.375534    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ab09ae-serving-cert\""
Oct 30 03:06:18.420278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.420215    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jwb\" (UniqueName: \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.420526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.420485    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.420702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.420679    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.420864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.420840    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.522691    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jwb\" (UniqueName: \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.522939    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.523116    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.523470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.529355    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.531252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.530609    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.573252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.572542    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.595315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.595175    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jwb\" (UniqueName: \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\") pod \"maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:18.688387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:18.687561    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:19.021439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:19.021412    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:06:19.022932 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:06:19.022906    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eb50bc_01ac_4b54_8b99_6042ef702914.slice/crio-fa6bf751d6ccd21af935ad6fb34ec514dabd3f601c5c56c5c73ab738fe8a4941 WatchSource:0}: Error finding container fa6bf751d6ccd21af935ad6fb34ec514dabd3f601c5c56c5c73ab738fe8a4941: Status 404 returned error can't find the container with id fa6bf751d6ccd21af935ad6fb34ec514dabd3f601c5c56c5c73ab738fe8a4941
Oct 30 03:06:20.034712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:20.034628    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" event={"ID":"56eb50bc-01ac-4b54-8b99-6042ef702914","Type":"ContainerStarted","Data":"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"}
Oct 30 03:06:20.034712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:20.034691    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" event={"ID":"56eb50bc-01ac-4b54-8b99-6042ef702914","Type":"ContainerStarted","Data":"fa6bf751d6ccd21af935ad6fb34ec514dabd3f601c5c56c5c73ab738fe8a4941"}
Oct 30 03:06:20.035304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:20.034733    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:20.077531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:20.077481    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" podStartSLOduration=1.31703218 podStartE2EDuration="2.077466642s" totalImagesPullingTime="760.434462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:20.074539108 +0000 UTC m=+1519.148709903" watchObservedRunningTime="2026-10-30 03:06:20.077466642 +0000 UTC m=+1519.151637435"
Oct 30 03:06:22.026250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.023787    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 03:06:22.026250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.023859    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 03:06:22.086112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.085657    4214 generic.go:356] "Generic (PLEG): container finished" podID="da558758-44e3-4838-ae46-7a5c1da799ab" containerID="a339793b6d54cc6d352473a811285a1bbe0eefce70a0965086b54e3d764983b1" exitCode=0
Oct 30 03:06:22.086112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.085778    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" event={"ID":"da558758-44e3-4838-ae46-7a5c1da799ab","Type":"ContainerDied","Data":"a339793b6d54cc6d352473a811285a1bbe0eefce70a0965086b54e3d764983b1"}
Oct 30 03:06:22.089617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.089114    4214 generic.go:356] "Generic (PLEG): container finished" podID="41cc5e10-e962-4b86-8902-b986982ead92" containerID="8f61cbedeae0d3ea5051626651f80d71d7cf23f78d6206b4c1e1988e79c60b35" exitCode=0
Oct 30 03:06:22.089617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.089219    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" event={"ID":"41cc5e10-e962-4b86-8902-b986982ead92","Type":"ContainerDied","Data":"8f61cbedeae0d3ea5051626651f80d71d7cf23f78d6206b4c1e1988e79c60b35"}
Oct 30 03:06:22.091647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.091338    4214 generic.go:356] "Generic (PLEG): container finished" podID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerID="5db97cb1864fa4951c0958cdff2a3462a2ba54c35f3baf25644e4ce42045fc6e" exitCode=0
Oct 30 03:06:22.091647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.091478    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" event={"ID":"b7d3d60e-10a7-4106-b959-ebcbc55ce921","Type":"ContainerDied","Data":"5db97cb1864fa4951c0958cdff2a3462a2ba54c35f3baf25644e4ce42045fc6e"}
Oct 30 03:06:22.094274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.093642    4214 generic.go:356] "Generic (PLEG): container finished" podID="c41471dd-1cb4-473c-8615-bc63682802dc" containerID="4c3c289cce4e6d30ea13825703128e3e3b6ec1405cf89d523bcc15ede0bc4a5c" exitCode=0
Oct 30 03:06:22.094274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.093714    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerDied","Data":"4c3c289cce4e6d30ea13825703128e3e3b6ec1405cf89d523bcc15ede0bc4a5c"}
Oct 30 03:06:22.094274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.093766    4214 scope.go:122] "RemoveContainer" containerID="3eb0fa0927ea5589edaa431107642a542ca14f919c4aba398859b4e9a43e8ce0"
Oct 30 03:06:22.216014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.215944    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 03:06:22.258457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.258433    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 03:06:22.263332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.263310    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 03:06:22.269073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.269049    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 03:06:22.324845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.324815    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.324908    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.324949    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.324975    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.325005    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.325030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.325143    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.325221    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.325368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.325288    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\") pod \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\" (UID: \"b7d3d60e-10a7-4106-b959-ebcbc55ce921\") "
Oct 30 03:06:22.326046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.326018    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.326277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.326146    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.326540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.326440    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.327366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.327338    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.327512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.327400    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:22.329374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.329351    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.331890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.331863    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:22.331993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.331909    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "kube-api-access-pfjzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.332062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.332043    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token" pod "b7d3d60e-10a7-4106-b959-ebcbc55ce921" (UID: "b7d3d60e-10a7-4106-b959-ebcbc55ce921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.426990 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.426954    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.427341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427318    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.427592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427575    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.427733 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427719    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.427848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427835    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.428013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427998    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.428127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428113    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.428287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428271    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.428436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428422    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.428558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428543    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.428698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428682    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.428874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428858    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.429013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.428999    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.429142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429129    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.429324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429307    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.429444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429431    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.429551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429537    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.429707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429690    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.429867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429853    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.429993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429959    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.430139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430121    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\") pod \"41cc5e10-e962-4b86-8902-b986982ead92\" (UID: \"41cc5e10-e962-4b86-8902-b986982ead92\") "
Oct 30 03:06:22.430277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430263    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.430391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430377    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.430497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430484    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.430604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430592    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\") pod \"c41471dd-1cb4-473c-8615-bc63682802dc\" (UID: \"c41471dd-1cb4-473c-8615-bc63682802dc\") "
Oct 30 03:06:22.430698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430686    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.430868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430853    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\") pod \"da558758-44e3-4838-ae46-7a5c1da799ab\" (UID: \"da558758-44e3-4838-ae46-7a5c1da799ab\") "
Oct 30 03:06:22.431931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.431905    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjzz\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-kube-api-access-pfjzz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432035    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432134    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432212    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432314    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432394    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432510    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432599    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.432692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432680    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d3d60e-10a7-4106-b959-ebcbc55ce921-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.435032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427318    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.427851    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429137    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.429690    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.430736    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.431153    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.431410    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432094    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.432446    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.433450    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "kube-api-access-lbdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.433829    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.434777    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "kube-api-access-vm88v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.435153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.434995    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.437145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.435475    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.437145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.435971    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.437145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.436198    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.437145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.436723    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:22.439076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.439045    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.440772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.440729    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:22.444539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444513    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy" pod "41cc5e10-e962-4b86-8902-b986982ead92" (UID: "41cc5e10-e962-4b86-8902-b986982ead92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.444640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444545    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:22.444640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444614    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.444756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444651    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.444756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444656    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:06:22.444756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444733    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:06:22.444893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.444811    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc" pod "da558758-44e3-4838-ae46-7a5c1da799ab" (UID: "da558758-44e3-4838-ae46-7a5c1da799ab"). InnerVolumeSpecName "kube-api-access-7qcjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.446516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.446481    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token" pod "c41471dd-1cb4-473c-8615-bc63682802dc" (UID: "c41471dd-1cb4-473c-8615-bc63682802dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535024    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41471dd-1cb4-473c-8615-bc63682802dc-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535066    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da558758-44e3-4838-ae46-7a5c1da799ab-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535081    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535134    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535147    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535159    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535171    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535185    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535198    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cc5e10-e962-4b86-8902-b986982ead92-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535210    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535222    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535283    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535296    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da558758-44e3-4838-ae46-7a5c1da799ab-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535314    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535327    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qcjc\" (UniqueName: \"kubernetes.io/projected/da558758-44e3-4838-ae46-7a5c1da799ab-kube-api-access-7qcjc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535342    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41471dd-1cb4-473c-8615-bc63682802dc-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535356    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535368    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da558758-44e3-4838-ae46-7a5c1da799ab-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535379    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535394    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm88v\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-kube-api-access-vm88v\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535405    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cc5e10-e962-4b86-8902-b986982ead92-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535418    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535431    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cc5e10-e962-4b86-8902-b986982ead92-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535443    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cc5e10-e962-4b86-8902-b986982ead92-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535454    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535466    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdbn\" (UniqueName: \"kubernetes.io/projected/c41471dd-1cb4-473c-8615-bc63682802dc-kube-api-access-lbdbn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:22.535554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:22.535478    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41471dd-1cb4-473c-8615-bc63682802dc-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:23.107258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.107202    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"
Oct 30 03:06:23.107651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.107332    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" event={"ID":"c41471dd-1cb4-473c-8615-bc63682802dc","Type":"ContainerDied","Data":"743f111e0285108192b0efa70db393741df620d4586f7bec5a14de37cf571006"}
Oct 30 03:06:23.107651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.107402    4214 scope.go:122] "RemoveContainer" containerID="4c3c289cce4e6d30ea13825703128e3e3b6ec1405cf89d523bcc15ede0bc4a5c"
Oct 30 03:06:23.111063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.110636    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"
Oct 30 03:06:23.111063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.110669    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b" event={"ID":"da558758-44e3-4838-ae46-7a5c1da799ab","Type":"ContainerDied","Data":"ae35acc79610409013cca2e39a95c1ca43c0a49b6d76f514b80111ddd6852467"}
Oct 30 03:06:23.112973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.112918    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz" event={"ID":"41cc5e10-e962-4b86-8902-b986982ead92","Type":"ContainerDied","Data":"3bdb15a74ffe05074cce2e774f0315d84d3773a6b6fe71b7857cb1c8214ac6f4"}
Oct 30 03:06:23.113093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.113014    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"
Oct 30 03:06:23.115391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.115366    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr" event={"ID":"b7d3d60e-10a7-4106-b959-ebcbc55ce921","Type":"ContainerDied","Data":"c3eceab602ffb3a54e556037c3d569a7cc357fe80b454fca2fffd0630d3f01bd"}
Oct 30 03:06:23.115497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.115389    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"
Oct 30 03:06:23.122116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.122102    4214 scope.go:122] "RemoveContainer" containerID="a339793b6d54cc6d352473a811285a1bbe0eefce70a0965086b54e3d764983b1"
Oct 30 03:06:23.131688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.131673    4214 scope.go:122] "RemoveContainer" containerID="8f61cbedeae0d3ea5051626651f80d71d7cf23f78d6206b4c1e1988e79c60b35"
Oct 30 03:06:23.141714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.141693    4214 scope.go:122] "RemoveContainer" containerID="5db97cb1864fa4951c0958cdff2a3462a2ba54c35f3baf25644e4ce42045fc6e"
Oct 30 03:06:23.184692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.184664    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 03:06:23.189076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.189048    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-whbbr"]
Oct 30 03:06:23.222918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.222891    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 03:06:23.226972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.226945    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8rd4b"]
Oct 30 03:06:23.264943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.264914    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 03:06:23.266102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.266073    4214 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:06:23.266280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.266128    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:23.271826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.271801    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-z9xgv"]
Oct 30 03:06:23.300240 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.300209    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 03:06:23.305412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:23.305385    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-9cmqz"]
Oct 30 03:06:25.072490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:25.072454    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cc5e10-e962-4b86-8902-b986982ead92" path="/var/lib/kubelet/pods/41cc5e10-e962-4b86-8902-b986982ead92/volumes"
Oct 30 03:06:25.073123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:25.073106    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" path="/var/lib/kubelet/pods/b7d3d60e-10a7-4106-b959-ebcbc55ce921/volumes"
Oct 30 03:06:25.073624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:25.073607    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" path="/var/lib/kubelet/pods/c41471dd-1cb4-473c-8615-bc63682802dc/volumes"
Oct 30 03:06:25.074506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:25.074486    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" path="/var/lib/kubelet/pods/da558758-44e3-4838-ae46-7a5c1da799ab/volumes"
Oct 30 03:06:26.049485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:26.049457    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:06:26.913961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:26.913921    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:06:26.914466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:26.914197    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" podUID="1b4b021c-f644-47e4-b898-c8af7b36a808" containerName="maas-api" containerID="cri-o://f500f1fdbca8dc70eaeb911e87b878708b29a8843613e431fd7cab2ef3e4eff9" gracePeriod=30
Oct 30 03:06:27.220834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.219802    4214 generic.go:356] "Generic (PLEG): container finished" podID="1b4b021c-f644-47e4-b898-c8af7b36a808" containerID="f500f1fdbca8dc70eaeb911e87b878708b29a8843613e431fd7cab2ef3e4eff9" exitCode=0
Oct 30 03:06:27.220834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.219983    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" event={"ID":"1b4b021c-f644-47e4-b898-c8af7b36a808","Type":"ContainerDied","Data":"f500f1fdbca8dc70eaeb911e87b878708b29a8843613e431fd7cab2ef3e4eff9"}
Oct 30 03:06:27.401969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.401405    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:06:27.543996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.543960    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\") pod \"1b4b021c-f644-47e4-b898-c8af7b36a808\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") "
Oct 30 03:06:27.544295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.544275    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\") pod \"1b4b021c-f644-47e4-b898-c8af7b36a808\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") "
Oct 30 03:06:27.544513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.544497    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") pod \"1b4b021c-f644-47e4-b898-c8af7b36a808\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") "
Oct 30 03:06:27.544724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.544710    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\" (UniqueName: \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\") pod \"1b4b021c-f644-47e4-b898-c8af7b36a808\" (UID: \"1b4b021c-f644-47e4-b898-c8af7b36a808\") "
Oct 30 03:06:27.581501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.581468    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca" pod "1b4b021c-f644-47e4-b898-c8af7b36a808" (UID: "1b4b021c-f644-47e4-b898-c8af7b36a808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:27.603689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.601451    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc" pod "1b4b021c-f644-47e4-b898-c8af7b36a808" (UID: "1b4b021c-f644-47e4-b898-c8af7b36a808"). InnerVolumeSpecName "kube-api-access-4d8zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:27.614043 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.608751    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs" pod "1b4b021c-f644-47e4-b898-c8af7b36a808" (UID: "1b4b021c-f644-47e4-b898-c8af7b36a808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:06:27.652317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.641928    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls" pod "1b4b021c-f644-47e4-b898-c8af7b36a808" (UID: "1b4b021c-f644-47e4-b898-c8af7b36a808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:06:27.676258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.672632    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:27.676258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.672673    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8zc\" (UniqueName: \"kubernetes.io/projected/1b4b021c-f644-47e4-b898-c8af7b36a808-kube-api-access-4d8zc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:27.676258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.672691    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b021c-f644-47e4-b898-c8af7b36a808-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:27.676258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:27.672705    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b4b021c-f644-47e4-b898-c8af7b36a808-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:28.327305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.321854    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw" event={"ID":"1b4b021c-f644-47e4-b898-c8af7b36a808","Type":"ContainerDied","Data":"8997a11b685e88ecd6665e67f71e129ce03310cae4c3244c9d5d0b177012e3b3"}
Oct 30 03:06:28.327305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.321929    4214 scope.go:122] "RemoveContainer" containerID="f500f1fdbca8dc70eaeb911e87b878708b29a8843613e431fd7cab2ef3e4eff9"
Oct 30 03:06:28.327305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.322094    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"
Oct 30 03:06:28.541255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.533267    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:06:28.561252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.556621    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-96wfw"]
Oct 30 03:06:28.748306 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:28.747645    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b021c_f644_47e4_b898_c8af7b36a808.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 03:06:28.776264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.775329    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:28.858265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.847352    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:28.885253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:28.875407    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:06:29.120253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:29.120064    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4b021c-f644-47e4-b898-c8af7b36a808" path="/var/lib/kubelet/pods/1b4b021c-f644-47e4-b898-c8af7b36a808/volumes"
Oct 30 03:06:30.834254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:30.833305    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:06:30.834754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:30.834341    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:30.963254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:30.962304    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:31.117312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.111665    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:31.683251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.682318    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:31.709254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.708317    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:31.919253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.889159    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:06:31.920124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.894903    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:06:31.920878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.920855    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:06:31.921069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.921054    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:06:31.997401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.997309    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:31.997662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:31.997641    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:06:32.125254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.124221    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:06:32.125254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.124740    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:06:32.580909 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.580879    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:06:32.663472 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.634104    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:06:32.715403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.714791    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:06:32.730253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:32.729399    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:06:35.934479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.934435    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:06:35.935757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935730    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy"
Oct 30 03:06:35.935757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935758    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935778    4214 cpu_manager.go:414] "containerMap: removing container" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935787    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935802    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935810    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935848    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1b4b021c-f644-47e4-b898-c8af7b36a808" containerName="maas-api"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935856    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4b021c-f644-47e4-b898-c8af7b36a808" containerName="maas-api"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935868    4214 cpu_manager.go:414] "containerMap: removing container" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935876    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935891    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:35.935894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.935899    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:35.936335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.936087    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4b021c-f644-47e4-b898-c8af7b36a808" containerName="maas-api"
Oct 30 03:06:35.936335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.936098    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="da558758-44e3-4838-ae46-7a5c1da799ab" containerName="istio-proxy"
Oct 30 03:06:35.936335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.936113    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:35.936335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.936133    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d3d60e-10a7-4106-b959-ebcbc55ce921" containerName="istio-proxy"
Oct 30 03:06:35.936335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.936146    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cc5e10-e962-4b86-8902-b986982ead92" containerName="istio-proxy"
Oct 30 03:06:35.941820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.941797    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:35.970828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.970775    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cea203-serving-cert\""
Oct 30 03:06:35.975454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:35.973505    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:06:36.054953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.054910    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.055129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.054968    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.055223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.055150    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9px\" (UniqueName: \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.055341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.055302    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.177364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.176267    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9px\" (UniqueName: \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.177364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.176437    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.177364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.176547    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.177364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.176591    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.190308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.180989    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.220251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.211913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.220251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.212492    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.220251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.215341    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9px\" (UniqueName: \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\") pod \"maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.272456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.272409    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:36.643651 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:06:36.643606    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6960882_4dc2_45b0_9257_ec45a4714ce5.slice/crio-dc00f55c6b367be12c5d097265d3b41e64cc81fc4d8b109626b4620cb2ef95f1 WatchSource:0}: Error finding container dc00f55c6b367be12c5d097265d3b41e64cc81fc4d8b109626b4620cb2ef95f1: Status 404 returned error can't find the container with id dc00f55c6b367be12c5d097265d3b41e64cc81fc4d8b109626b4620cb2ef95f1
Oct 30 03:06:36.644632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:36.644594    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:06:37.564254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.563423    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:06:37.564254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.563491    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:06:37.623376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.623332    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" event={"ID":"b6960882-4dc2-45b0-9257-ec45a4714ce5","Type":"ContainerStarted","Data":"d67527f1e2738b612dfd4a452f87b00734c3fabdda3b7e801f4dae1f7100c54d"}
Oct 30 03:06:37.623620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.623443    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" event={"ID":"b6960882-4dc2-45b0-9257-ec45a4714ce5","Type":"ContainerStarted","Data":"dc00f55c6b367be12c5d097265d3b41e64cc81fc4d8b109626b4620cb2ef95f1"}
Oct 30 03:06:37.623909 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.623887    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:37.673005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:37.672941    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" podStartSLOduration=2.35349619 podStartE2EDuration="2.67292289s" totalImagesPullingTime="319.4267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:37.666413779 +0000 UTC m=+1536.740584564" watchObservedRunningTime="2026-10-30 03:06:37.67292289 +0000 UTC m=+1536.747093684"
Oct 30 03:06:38.670446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.670419    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:06:38.671452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.671430    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41471dd-1cb4-473c-8615-bc63682802dc" containerName="istio-proxy"
Oct 30 03:06:38.674505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.674483    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:38.693582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.690541    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-be9f2a59-dockercfg-vw9wl\""
Oct 30 03:06:38.694364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.694313    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:06:38.705556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.703724    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-be9f2a59\""
Oct 30 03:06:38.934306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.933906    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:38.934306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:38.934056    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vpq\" (UniqueName: \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.039351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.039318    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:06:39.057734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.055483    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vpq\" (UniqueName: \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.057734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.055838    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.057734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.057249    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.061824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.061201    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.084540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.084494    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:06:39.086216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.086190    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vpq\" (UniqueName: \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.171338 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:39.171297    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:06:39.258548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.258461    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.258548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.258500    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rh6\" (UniqueName: \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.297159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.297125    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:39.361437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.360016    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.361437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.360074    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rh6\" (UniqueName: \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.361713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.361691    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.381620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.381594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rh6\" (UniqueName: \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.387292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.387271    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:39.607489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.607458    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:06:39.607801 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:06:39.607781    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9a0261_e710_484a_a436_2a7befa7190a.slice/crio-1e8efdd5fa698e8567a06603cb7831b7fdd50605f46fe53093fe154271438bde WatchSource:0}: Error finding container 1e8efdd5fa698e8567a06603cb7831b7fdd50605f46fe53093fe154271438bde: Status 404 returned error can't find the container with id 1e8efdd5fa698e8567a06603cb7831b7fdd50605f46fe53093fe154271438bde
Oct 30 03:06:39.647593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.647555    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" event={"ID":"6a9a0261-e710-484a-a436-2a7befa7190a","Type":"ContainerStarted","Data":"1e8efdd5fa698e8567a06603cb7831b7fdd50605f46fe53093fe154271438bde"}
Oct 30 03:06:39.650010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.649986    4214 generic.go:356] "Generic (PLEG): container finished" podID="853fa3d3-0a35-4181-9432-405ba12d0589" containerID="462cd71628422504745e1ac26b3ceb22037dc5970698c42314c7d874cf994495" exitCode=0
Oct 30 03:06:39.650136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.650055    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerDied","Data":"462cd71628422504745e1ac26b3ceb22037dc5970698c42314c7d874cf994495"}
Oct 30 03:06:39.650546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.650530    4214 scope.go:122] "RemoveContainer" containerID="462cd71628422504745e1ac26b3ceb22037dc5970698c42314c7d874cf994495"
Oct 30 03:06:39.652912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.652871    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:39.653001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.652960    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:39.653046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.653035    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:39.727217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:39.727172    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:06:40.674254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.672547    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" event={"ID":"6a9a0261-e710-484a-a436-2a7befa7190a","Type":"ContainerStarted","Data":"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"}
Oct 30 03:06:40.676020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.675708    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerStarted","Data":"f57b3b4f031548c0a48d2aa9dce389e9ef66130a25b4fb8f9286b0ab54ff99f0"}
Oct 30 03:06:40.679106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.679077    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" event={"ID":"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23","Type":"ContainerStarted","Data":"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"}
Oct 30 03:06:40.679296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.679126    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" event={"ID":"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23","Type":"ContainerStarted","Data":"1649dcb7d898728a1d02e611656e5fc8ba34e488efdc066494af4b4e6df46395"}
Oct 30 03:06:40.679296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.679185    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:40.719362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.718903    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" podStartSLOduration=2.718884588 podStartE2EDuration="2.718884588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:40.715571818 +0000 UTC m=+1539.789742605" watchObservedRunningTime="2026-10-30 03:06:40.718884588 +0000 UTC m=+1539.793055382"
Oct 30 03:06:40.801414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:40.801350    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" podStartSLOduration=2.801331701 podStartE2EDuration="2.801331701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:40.798023686 +0000 UTC m=+1539.872194479" watchObservedRunningTime="2026-10-30 03:06:40.801331701 +0000 UTC m=+1539.875502495"
Oct 30 03:06:41.192672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.189680    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:06:41.196407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.195184    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.204248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.203178    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:06:41.204735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.204710    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:06:41.204854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.204742    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:06:41.204854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.204803    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:06:41.212970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.212935    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213001    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213050    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213120    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213191    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z448d\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213277    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213401    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213454    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.213525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.213492    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.219188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.219164    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-71f552-openshift-default-dockercfg-xft8b\""
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318285    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318369    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318414    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318485    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318567    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z448d\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318641    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318771    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318825    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.318866    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.356694    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.377256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.356956    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.402279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.380282    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.490323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.383595    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.490323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.477114    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z448d\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.516778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.516742    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.564607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.529042    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.659344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.634164    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.659344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.634424    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\") pod \"e2e-api-b-71f552-openshift-default-665776fb8c-8j94q\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") " pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:41.802810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.802782    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:41.829240 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:41.829202    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:42.214336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.212167    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:06:42.214336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.212242    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:06:42.223956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.223786    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:06:42.227354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.227324    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:42.227436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.227392    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:42.227436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.227420    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:06:42.812287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.812251    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerStarted","Data":"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"}
Oct 30 03:06:42.812737 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.812320    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerStarted","Data":"0afce980f5bcbce181e506a11aa0d8a4afd7a6303d07c1281c308aa7c40816a6"}
Oct 30 03:06:42.829268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.829244    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:42.885325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:42.885261    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podStartSLOduration=1.8852422610000001 podStartE2EDuration="1.885242261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:42.879040563 +0000 UTC m=+1541.953211362" watchObservedRunningTime="2026-10-30 03:06:42.885242261 +0000 UTC m=+1541.959413052"
Oct 30 03:06:43.204924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.204826    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:06:43.204924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.204902    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:06:43.671257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.668424    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:06:43.864251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.856510    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:43.883532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.883145    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:43.884437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:43.884416    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:06:44.227260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:44.226013    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:06:44.227260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:44.226075    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:06:45.205060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:45.205026    4214 patch_prober.go:29] interesting pod/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:06:45.205509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:45.205078    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:06:46.373743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:46.373709    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:06:46.374670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:46.374641    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:06:46.375996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:46.375973    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:06:47.047861 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.047824    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:06:47.052986 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.052956    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.083735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.083710    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-71f552\""
Oct 30 03:06:47.090385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.090362    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:06:47.155499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.155466    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:06:47.159336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.159314    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.217528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.212645    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-71f552-dockercfg-cwhcd\""
Oct 30 03:06:47.271494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.271457    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.271844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.271824    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfjc\" (UniqueName: \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.272333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.272306    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.272504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.272488    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgtk\" (UniqueName: \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.434548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.420472    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:06:47.466279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.466211    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.466566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.466536    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgtk\" (UniqueName: \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.466797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.466768    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.467033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.467008    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfjc\" (UniqueName: \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.469311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.469269    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.469479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.469447    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.527584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.527550    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgtk\" (UniqueName: \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\") pod \"payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") " pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.533569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.533532    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfjc\" (UniqueName: \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\") pod \"payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:47.671353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.671316    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:47.787686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:47.787654    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:48.000396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.000369    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:06:48.000513 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:06:48.000418    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c36f5c_ce33_464e_940d_29bef9639b7e.slice/crio-e5eac79ed90ac932a3284f98be151072d198cef5bb4f372c1dfb1af028e0e759 WatchSource:0}: Error finding container e5eac79ed90ac932a3284f98be151072d198cef5bb4f372c1dfb1af028e0e759: Status 404 returned error can't find the container with id e5eac79ed90ac932a3284f98be151072d198cef5bb4f372c1dfb1af028e0e759
Oct 30 03:06:48.128428 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:06:48.128402    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d100648_f008_4989_90c2_4cc43ac4bb67.slice/crio-2bea9a3874ff054c764b8801513eae5cc4b811b3f9e964524206ac5782bf219e WatchSource:0}: Error finding container 2bea9a3874ff054c764b8801513eae5cc4b811b3f9e964524206ac5782bf219e: Status 404 returned error can't find the container with id 2bea9a3874ff054c764b8801513eae5cc4b811b3f9e964524206ac5782bf219e
Oct 30 03:06:48.128945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.128921    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:06:48.929708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.929668    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" event={"ID":"5d100648-f008-4989-90c2-4cc43ac4bb67","Type":"ContainerStarted","Data":"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"}
Oct 30 03:06:48.930163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.929732    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" event={"ID":"5d100648-f008-4989-90c2-4cc43ac4bb67","Type":"ContainerStarted","Data":"2bea9a3874ff054c764b8801513eae5cc4b811b3f9e964524206ac5782bf219e"}
Oct 30 03:06:48.930163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.929760    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:06:48.931701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.931674    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" event={"ID":"b5c36f5c-ce33-464e-940d-29bef9639b7e","Type":"ContainerStarted","Data":"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"}
Oct 30 03:06:48.931809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.931720    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:06:48.931809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.931738    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" event={"ID":"b5c36f5c-ce33-464e-940d-29bef9639b7e","Type":"ContainerStarted","Data":"e5eac79ed90ac932a3284f98be151072d198cef5bb4f372c1dfb1af028e0e759"}
Oct 30 03:06:48.971755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:48.971706    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" podStartSLOduration=1.9716933110000001 podStartE2EDuration="1.971693311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:48.969670084 +0000 UTC m=+1548.043840878" watchObservedRunningTime="2026-10-30 03:06:48.971693311 +0000 UTC m=+1548.045864105"
Oct 30 03:06:49.016122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:49.016073    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" podStartSLOduration=3.016047091 podStartE2EDuration="3.016047091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:49.008913661 +0000 UTC m=+1548.083084454" watchObservedRunningTime="2026-10-30 03:06:49.016047091 +0000 UTC m=+1548.090217923"
Oct 30 03:06:51.688444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:51.688400    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:06:51.807106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:51.807078    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:06:52.815672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:52.815644    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:06:56.689030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:56.688974    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" podUID="7381385e-3142-47e0-a688-2ca263b966c9" containerName="payload-pre-processing" containerID="cri-o://a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f" gracePeriod=25
Oct 30 03:06:57.246389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.246043    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:06:57.254614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.250652    4214 generic.go:356] "Generic (PLEG): container finished" podID="7381385e-3142-47e0-a688-2ca263b966c9" containerID="a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f" exitCode=0
Oct 30 03:06:57.254614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.252055    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" event={"ID":"7381385e-3142-47e0-a688-2ca263b966c9","Type":"ContainerDied","Data":"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"}
Oct 30 03:06:57.254614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.252109    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d" event={"ID":"7381385e-3142-47e0-a688-2ca263b966c9","Type":"ContainerDied","Data":"8fa57b942ea759f156d54892b93b6b7900fde3c4d3186318df9270555cfe2486"}
Oct 30 03:06:57.254614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.252139    4214 scope.go:122] "RemoveContainer" containerID="a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"
Oct 30 03:06:57.274109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.274080    4214 scope.go:122] "RemoveContainer" containerID="a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"
Oct 30 03:06:57.274552 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:06:57.274528    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f\": container with ID starting with a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f not found: ID does not exist" containerID="a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"
Oct 30 03:06:57.274675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.274565    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f"} err="failed to get container status \"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f\": rpc error: code = NotFound desc = could not find container \"a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f\": container with ID starting with a596982ca6f597b6a8370d1a0e59049b120e4c0c12dcf15e381ab11f1e01fb4f not found: ID does not exist"
Oct 30 03:06:57.367738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.367707    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\") pod \"7381385e-3142-47e0-a688-2ca263b966c9\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") "
Oct 30 03:06:57.368359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.368261    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume" pod "7381385e-3142-47e0-a688-2ca263b966c9" (UID: "7381385e-3142-47e0-a688-2ca263b966c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:57.368492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.368428    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\" (UniqueName: \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\") pod \"7381385e-3142-47e0-a688-2ca263b966c9\" (UID: \"7381385e-3142-47e0-a688-2ca263b966c9\") "
Oct 30 03:06:57.369279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.369251    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7381385e-3142-47e0-a688-2ca263b966c9-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:57.370704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.370677    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2" pod "7381385e-3142-47e0-a688-2ca263b966c9" (UID: "7381385e-3142-47e0-a688-2ca263b966c9"). InnerVolumeSpecName "kube-api-access-9tks2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:57.471186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:57.471150    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tks2\" (UniqueName: \"kubernetes.io/projected/7381385e-3142-47e0-a688-2ca263b966c9-kube-api-access-9tks2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:06:58.264603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:58.264576    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"
Oct 30 03:06:58.320827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:58.320801    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:06:58.335966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:58.335934    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ffb1886-865ccb8879-d8h6d"]
Oct 30 03:06:59.071030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:59.070984    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7381385e-3142-47e0-a688-2ca263b966c9" path="/var/lib/kubelet/pods/7381385e-3142-47e0-a688-2ca263b966c9/volumes"
Oct 30 03:06:59.995611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:06:59.995458    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:07:00.014253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:00.007890    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:07:04.906960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:04.906780    4214 scope.go:122] "RemoveContainer" containerID="bf220a573158ea181b1f6f67b29cc7ce7fc1ac9bd44b8f54b9df77103bba30bb"
Oct 30 03:07:04.921071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:04.921047    4214 scope.go:122] "RemoveContainer" containerID="be06d23b8d5e9788e88c35430c32b8dee0e5a9a2377e813e27c83a3ada243465"
Oct 30 03:07:06.029956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:06.029881    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"]
Oct 30 03:07:06.045348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:06.045321    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a273f80-gh5v6"]
Oct 30 03:07:07.079099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:07.079048    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121b90cb-544e-4a6d-ba94-74937c0c865a" path="/var/lib/kubelet/pods/121b90cb-544e-4a6d-ba94-74937c0c865a/volumes"
Oct 30 03:07:09.179061 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:07:09.179015    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:07:09.272409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:09.272373    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:07:09.272628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:09.272451    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:10.490873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.490788    4214 generic.go:356] "Generic (PLEG): container finished" podID="2f636458-995c-42ef-a64e-a2fd496443fa" containerID="a566b47bd0b7e0d3c4f1853293de38d76de54c412280401fbaa29be100fbb4ee" exitCode=0
Oct 30 03:07:10.491372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.490869    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerDied","Data":"a566b47bd0b7e0d3c4f1853293de38d76de54c412280401fbaa29be100fbb4ee"}
Oct 30 03:07:10.491445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.491408    4214 scope.go:122] "RemoveContainer" containerID="a566b47bd0b7e0d3c4f1853293de38d76de54c412280401fbaa29be100fbb4ee"
Oct 30 03:07:10.492735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.492714    4214 generic.go:356] "Generic (PLEG): container finished" podID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerID="22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171" exitCode=0
Oct 30 03:07:10.492810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.492792    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerDied","Data":"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"}
Oct 30 03:07:10.493207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.493191    4214 scope.go:122] "RemoveContainer" containerID="22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"
Oct 30 03:07:10.493806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.493722    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:10.493849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.493831    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:10.493898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.493881    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:10.494968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.494937    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:10.495066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.495014    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:10.495066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:10.495051    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:11.548250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:11.540191    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerStarted","Data":"43713bd079a64fc04b772de647a78085be5532e2cba331cc8834cbefd230569a"}
Oct 30 03:07:11.548250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:11.545986    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerStarted","Data":"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"}
Oct 30 03:07:11.670255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:11.669442    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:07:11.670255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:11.669927    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:11.670255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:11.669997    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:12.311525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:12.311487    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:07:12.311824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:12.311720    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:12.311935 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:12.311846    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:12.668704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:12.668607    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:12.669329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:12.669294    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:13.312597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.312559    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:13.312769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.312630    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:13.482253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.480696    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:07:13.482253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.482116    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7381385e-3142-47e0-a688-2ca263b966c9" containerName="payload-pre-processing"
Oct 30 03:07:13.482253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.482140    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7381385e-3142-47e0-a688-2ca263b966c9" containerName="payload-pre-processing"
Oct 30 03:07:13.482658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.482331    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7381385e-3142-47e0-a688-2ca263b966c9" containerName="payload-pre-processing"
Oct 30 03:07:13.515611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.511311    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:07:13.515611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.511444    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.530596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.530564    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.530793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.530776    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8np4\" (UniqueName: \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.530871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.530809    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.530938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.530911    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.533766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.533739    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-be9f2a59-serving-cert\""
Oct 30 03:07:13.633479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.633361    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8np4\" (UniqueName: \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.633788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.633763    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.676303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.645591    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.676303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.645900    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.676303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.646800    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.678015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.677576    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:13.678015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.677632    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:13.704976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.703932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8np4\" (UniqueName: \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.705179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.705149    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.708485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.708385    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\") pod \"maas-api-e2e-mt-be9f2a59-697585b688-dlb6b\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:13.896335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:13.888350    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:14.312629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.312589    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:14.312789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.312659    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:14.498943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.496893    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:07:14.669247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.668594    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:14.669247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.668652    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:14.692250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:14.687927    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" event={"ID":"095850da-0b02-4df3-b98c-a3c450e5eac3","Type":"ContainerStarted","Data":"56eee315b316a868af962d5203e11b6be97127357cf17e1acfaef26360772429"}
Oct 30 03:07:15.317256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.313791    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:15.317256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.313864    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:15.665930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.665624    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:15.666209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.666178    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:15.777756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.777713    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" event={"ID":"095850da-0b02-4df3-b98c-a3c450e5eac3","Type":"ContainerStarted","Data":"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"}
Oct 30 03:07:15.779958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.779932    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:15.874256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:15.857846    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" podStartSLOduration=2.420934947 podStartE2EDuration="2.857824144s" totalImagesPullingTime="436.889197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:15.833059777 +0000 UTC m=+1574.907230594" watchObservedRunningTime="2026-10-30 03:07:15.857824144 +0000 UTC m=+1574.931994940"
Oct 30 03:07:16.321267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:16.316826    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:16.321267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:16.316889    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:16.670311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:16.669672    4214 patch_prober.go:29] interesting pod/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 03:07:16.670311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:16.669736    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 03:07:17.312552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:17.312503    4214 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 03:07:17.312974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:17.312589    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 03:07:17.671133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:17.671032    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:07:17.671823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:17.671799    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:07:17.674619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:17.673642    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:07:18.379292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:18.379142    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:07:18.381503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:18.381437    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:07:18.389342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:18.385457    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:07:22.834244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:22.834197    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:07:29.248733 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.248696    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:07:29.252648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.252629    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299301    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299411    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5gv\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299487    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299541    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299613    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299674    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299757    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.299942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.299903    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.302028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.301995    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:07:29.302184 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.302018    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-67440d-openshift-default-dockercfg-qxlc2\""
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427306    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427421    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427555    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5gv\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427707    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427788    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427840    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427874    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.427956    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.439271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.428064    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.452248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.443897    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.452248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.444166    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.452248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.445186    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.452248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.445562    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.460250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.453815    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.460250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.456195    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.474355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.472727    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.612252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.610987    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.613627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.613567    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5gv\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\") pod \"e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") " pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:29.866933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:29.866841    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:30.222830 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:07:30.222793    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844932dc_68d5_495b_9be3_b9023e0ba1be.slice/crio-52e117dd91e39d0866efcc2d6f1ea8f2d791df10694f5ff1488507eb8a001e42 WatchSource:0}: Error finding container 52e117dd91e39d0866efcc2d6f1ea8f2d791df10694f5ff1488507eb8a001e42: Status 404 returned error can't find the container with id 52e117dd91e39d0866efcc2d6f1ea8f2d791df10694f5ff1488507eb8a001e42
Oct 30 03:07:30.226222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:30.226186    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:30.226343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:30.226308    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:30.226405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:30.226348    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:07:30.233308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:30.233284    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:07:31.120026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.119272    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerStarted","Data":"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"}
Oct 30 03:07:31.120026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.119342    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerStarted","Data":"52e117dd91e39d0866efcc2d6f1ea8f2d791df10694f5ff1488507eb8a001e42"}
Oct 30 03:07:31.199473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.199419    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podStartSLOduration=2.199402393 podStartE2EDuration="2.199402393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:31.186153365 +0000 UTC m=+1590.260324224" watchObservedRunningTime="2026-10-30 03:07:31.199402393 +0000 UTC m=+1590.273573185"
Oct 30 03:07:31.208577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.208549    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:07:31.212482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.212461    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.257961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.255489    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:07:31.278642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.278578    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptwl\" (UniqueName: \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.278892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.278662    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.278892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.278713    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.278892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.278860    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.305492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.305455    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-71f552-serving-cert\""
Oct 30 03:07:31.380150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.380053    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptwl\" (UniqueName: \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.380150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.380101    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.380150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.380139    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.380496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.380271    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.380895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.380872    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.382901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.382877    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.383036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.383007    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.405182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.405131    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptwl\" (UniqueName: \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\") pod \"maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.552251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.551528    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:31.724388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.724069    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:07:31.728446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.728422    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.755097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.751559    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d30e9f\"/\"kube-root-ca.crt\""
Oct 30 03:07:31.755097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.751665    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d30e9f\"/\"default-dockercfg-vvm2w\""
Oct 30 03:07:31.755097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.751733    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d30e9f\"/\"test-model-d30e9f-kserve-self-signed-certs\""
Oct 30 03:07:31.756851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.756243    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d30e9f\"/\"openshift-service-ca.crt\""
Oct 30 03:07:31.767055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.767029    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:07:31.808825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.808785    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzjl\" (UniqueName: \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.808988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.808849    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.809112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.809085    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.809161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.809141    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.809221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.809203    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.809576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.809557    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.867437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.867398    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:31.874081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.874054    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:31.911468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzjl\" (UniqueName: \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.911468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911423    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.911840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911619    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.911840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911658    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.911840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911685    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.912004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911882    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.912004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.911972    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.912124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.912102    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.912314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.912287    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.913962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.913943    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.914142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.914126    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.939519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.939489    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzjl\" (UniqueName: \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\") pod \"test-model-d30e9f-kserve-58c4b94c89-mfwzn\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") " pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:31.943880 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:07:31.943854    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9082263c_60c8_4cb4_9932_a1f626544281.slice/crio-73e82ae5eb9218e114a89b9127538c961237ce94d8445e9f1121182a4efed224 WatchSource:0}: Error finding container 73e82ae5eb9218e114a89b9127538c961237ce94d8445e9f1121182a4efed224: Status 404 returned error can't find the container with id 73e82ae5eb9218e114a89b9127538c961237ce94d8445e9f1121182a4efed224
Oct 30 03:07:31.944423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:31.944404    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:07:32.069387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:32.060353    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:32.437257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:32.419298    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" event={"ID":"9082263c-60c8-4cb4-9932-a1f626544281","Type":"ContainerStarted","Data":"73e82ae5eb9218e114a89b9127538c961237ce94d8445e9f1121182a4efed224"}
Oct 30 03:07:32.437257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:32.421223    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:32.437257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:32.422345    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:07:32.907884 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:07:32.907740    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b09d75b_1ca8_4cf1_9526_069f23e20138.slice/crio-129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc WatchSource:0}: Error finding container 129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc: Status 404 returned error can't find the container with id 129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc
Oct 30 03:07:32.908557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:32.908530    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:07:33.426977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:33.426938    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" event={"ID":"0b09d75b-1ca8-4cf1-9526-069f23e20138","Type":"ContainerStarted","Data":"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"}
Oct 30 03:07:33.427154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:33.426990    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" event={"ID":"0b09d75b-1ca8-4cf1-9526-069f23e20138","Type":"ContainerStarted","Data":"129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc"}
Oct 30 03:07:33.428619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:33.428587    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" event={"ID":"9082263c-60c8-4cb4-9932-a1f626544281","Type":"ContainerStarted","Data":"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"}
Oct 30 03:07:33.492760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:33.492704    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podStartSLOduration=2.492688357 podStartE2EDuration="2.492688357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:33.490218593 +0000 UTC m=+1592.564389386" watchObservedRunningTime="2026-10-30 03:07:33.492688357 +0000 UTC m=+1592.566859179"
Oct 30 03:07:33.573090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:33.573016    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" podStartSLOduration=2.005970685 podStartE2EDuration="2.572997542s" totalImagesPullingTime="567.026857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:33.565482834 +0000 UTC m=+1592.639653639" watchObservedRunningTime="2026-10-30 03:07:33.572997542 +0000 UTC m=+1592.647168336"
Oct 30 03:07:34.065387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:34.065356    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:34.519305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:34.517535    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:37.760178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.760137    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:07:37.765582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.765456    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:37.783676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.783639    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:07:37.787677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.787655    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-67440d\""
Oct 30 03:07:37.787847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.787671    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-67440d-dockercfg-fgcmx\""
Oct 30 03:07:37.813691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.813651    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:07:37.816380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.816343    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87vr\" (UniqueName: \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:37.816978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.816953    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:37.819371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.819345    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:37.839613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.839581    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:07:37.923272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:37.920014    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq95l\" (UniqueName: \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.009482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.009436    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87vr\" (UniqueName: \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:38.012037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.011962    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.012316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.012292    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:38.121277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.119100    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq95l\" (UniqueName: \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.121277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.119286    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.134266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.126217    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.173337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.162242    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:38.249966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.212892    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq95l\" (UniqueName: \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\") pod \"payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.249966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.217258    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87vr\" (UniqueName: \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\") pod \"payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") " pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:38.387600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.387569    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:38.449926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.442888    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:38.702938 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:07:38.702903    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfb82d5_1f38_41dd_9882_6068e161846f.slice/crio-4d9845ead71e0db6281d061ce8eb6a82271e96b4a1b37a11654f9ef3f917ede2 WatchSource:0}: Error finding container 4d9845ead71e0db6281d061ce8eb6a82271e96b4a1b37a11654f9ef3f917ede2: Status 404 returned error can't find the container with id 4d9845ead71e0db6281d061ce8eb6a82271e96b4a1b37a11654f9ef3f917ede2
Oct 30 03:07:38.703296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.703274    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:07:38.788190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:38.788164    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:07:38.793008 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:07:38.792980    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8aab19_1f5c_425f_a76f_d2e861657c88.slice/crio-05f53c480da49995a32ca91adb4160bc740be23a960e934beba23839f404c397 WatchSource:0}: Error finding container 05f53c480da49995a32ca91adb4160bc740be23a960e934beba23839f404c397: Status 404 returned error can't find the container with id 05f53c480da49995a32ca91adb4160bc740be23a960e934beba23839f404c397
Oct 30 03:07:39.186217 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:07:39.186180    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:07:39.588540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.588499    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" event={"ID":"adfb82d5-1f38-41dd-9882-6068e161846f","Type":"ContainerStarted","Data":"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"}
Oct 30 03:07:39.588736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.588589    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" event={"ID":"adfb82d5-1f38-41dd-9882-6068e161846f","Type":"ContainerStarted","Data":"4d9845ead71e0db6281d061ce8eb6a82271e96b4a1b37a11654f9ef3f917ede2"}
Oct 30 03:07:39.588815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.588791    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:39.590535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.590509    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" event={"ID":"7e8aab19-1f5c-425f-a76f-d2e861657c88","Type":"ContainerStarted","Data":"5d3cb410f708b5cfaa9cdee4377a566a0e2b4e56d72cc72e0b215a7bbdba5abc"}
Oct 30 03:07:39.590659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.590554    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" event={"ID":"7e8aab19-1f5c-425f-a76f-d2e861657c88","Type":"ContainerStarted","Data":"05f53c480da49995a32ca91adb4160bc740be23a960e934beba23839f404c397"}
Oct 30 03:07:39.590706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.590660    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:39.641423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.641359    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" podStartSLOduration=2.6413404910000002 podStartE2EDuration="2.641340491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:39.635670119 +0000 UTC m=+1598.709840912" watchObservedRunningTime="2026-10-30 03:07:39.641340491 +0000 UTC m=+1598.715511284"
Oct 30 03:07:39.689542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:39.689337    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" podStartSLOduration=2.689317972 podStartE2EDuration="2.689317972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:39.688836642 +0000 UTC m=+1598.763007435" watchObservedRunningTime="2026-10-30 03:07:39.689317972 +0000 UTC m=+1598.763488766"
Oct 30 03:07:40.515937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:40.515906    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:07:42.065440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:42.065402    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:42.079517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:42.079487    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:42.678598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:42.678570    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:43.439374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:43.439338    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:07:47.272253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.271614    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:07:47.367152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.367104    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy" containerID="cri-o://c15024eaa7a394354e0969593a31cee0c757ab49804b2e59b3106a6733b4184c" gracePeriod=30
Oct 30 03:07:47.411202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.411146    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:47.658385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.658338    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:47.687719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.687667    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:47.692831 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.692788    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:07:47.720259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.700593    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 30 03:07:47.720259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.719061    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:07:47.830338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:47.830290    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:07:48.105367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.082633    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:07:48.118627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.118582    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy" containerID="cri-o://e903d457c550f39e346e34049c4545362b6b1ba3c3f05651c0c877bc1fe02837" gracePeriod=30
Oct 30 03:07:48.389307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.388343    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xl95z" podUID="6ec5181a-fc48-4367-a7b6-b57fc2da69c1" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:48.458255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.455302    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" podUID="2aafd8ff-a536-42ec-9c86-446d2c05a767" containerName="payload-processing" containerID="cri-o://f045cb45e039b64138864ad4ad8b2cf537fe5a4ee74d3ae3c39e6ac0a3a1ee81" gracePeriod=25
Oct 30 03:07:48.560574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.557641    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:48.667312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.666442    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:07:48.667312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.667022    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:07:48.713254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.709771    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:48.713254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.710133    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": context deadline exceeded"
Oct 30 03:07:48.713254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.711742    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:07:48.713254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.711801    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:48.738794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.719674    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:48.738794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.720029    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:07:48.759892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.749303    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:48.759892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.749729    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:07:49.026693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.943436    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:49.026693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:48.943921    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:07:49.705252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.698368    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:07:49.712418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.710328    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:49.712418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.711191    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 30 03:07:49.734504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.734470    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:49.779124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.761888    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:07:49.958045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.957941    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:07:50.004254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:49.987029    4214 generic.go:356] "Generic (PLEG): container finished" podID="2aafd8ff-a536-42ec-9c86-446d2c05a767" containerID="f045cb45e039b64138864ad4ad8b2cf537fe5a4ee74d3ae3c39e6ac0a3a1ee81" exitCode=0
Oct 30 03:07:50.005367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.001744    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" event={"ID":"2aafd8ff-a536-42ec-9c86-446d2c05a767","Type":"ContainerDied","Data":"f045cb45e039b64138864ad4ad8b2cf537fe5a4ee74d3ae3c39e6ac0a3a1ee81"}
Oct 30 03:07:50.650812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.628278    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:07:50.652571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.652533    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:07:50.704265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.697204    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:07:50.704265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.698723    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:50.723758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.723710    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:50.780464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.780422    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:50.938327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.938241    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:07:51.025258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:50.997819    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:07:51.521099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.521065    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:07:51.529520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.529445    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb" event={"ID":"2aafd8ff-a536-42ec-9c86-446d2c05a767","Type":"ContainerDied","Data":"68a7a93470183e4be1e9d9b42bd2deebcd7ad8585c11afba6ac69a7e446a33e3"}
Oct 30 03:07:51.529874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.529839    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:07:51.530061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.530039    4214 scope.go:122] "RemoveContainer" containerID="f045cb45e039b64138864ad4ad8b2cf537fe5a4ee74d3ae3c39e6ac0a3a1ee81"
Oct 30 03:07:51.530551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.530202    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:07:51.530779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.530766    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:07:51.560725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.560688    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\") pod \"2aafd8ff-a536-42ec-9c86-446d2c05a767\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") "
Oct 30 03:07:51.560873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.560819    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\" (UniqueName: \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\") pod \"2aafd8ff-a536-42ec-9c86-446d2c05a767\" (UID: \"2aafd8ff-a536-42ec-9c86-446d2c05a767\") "
Oct 30 03:07:51.581767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.562104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume" pod "2aafd8ff-a536-42ec-9c86-446d2c05a767" (UID: "2aafd8ff-a536-42ec-9c86-446d2c05a767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:51.581767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.565755    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m" pod "2aafd8ff-a536-42ec-9c86-446d2c05a767" (UID: "2aafd8ff-a536-42ec-9c86-446d2c05a767"). InnerVolumeSpecName "kube-api-access-4wn5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:51.608869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.608829    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:07:51.682253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.679962    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aafd8ff-a536-42ec-9c86-446d2c05a767-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:51.682253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:51.679998    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wn5m\" (UniqueName: \"kubernetes.io/projected/2aafd8ff-a536-42ec-9c86-446d2c05a767-kube-api-access-4wn5m\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:52.533511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.533484    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"
Oct 30 03:07:52.569446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.568558    4214 generic.go:356] "Generic (PLEG): container finished" podID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerID="c15024eaa7a394354e0969593a31cee0c757ab49804b2e59b3106a6733b4184c" exitCode=0
Oct 30 03:07:52.569446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.568668    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" event={"ID":"7ff07ceb-8b79-4ccf-8b28-acce23859fcf","Type":"ContainerDied","Data":"c15024eaa7a394354e0969593a31cee0c757ab49804b2e59b3106a6733b4184c"}
Oct 30 03:07:52.726306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.723684    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:07:52.726860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.726841    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1abc2-75c7bd5cb4-kqvsb"]
Oct 30 03:07:52.893480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:52.893449    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:07:53.013115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013026    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013079    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013147    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013194    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013250    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013407    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013442    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013627    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.013710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.013678    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\") pod \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\" (UID: \"7ff07ceb-8b79-4ccf-8b28-acce23859fcf\") "
Oct 30 03:07:53.014278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.014250    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:53.014843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.014817    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:53.015946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.015919    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:53.016504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.016479    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:53.017450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.017428    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:53.022447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.022408    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:53.023103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.022971    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:07:53.023103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.023069    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "kube-api-access-42xwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:53.023409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.023372    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy" pod "7ff07ceb-8b79-4ccf-8b28-acce23859fcf" (UID: "7ff07ceb-8b79-4ccf-8b28-acce23859fcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:53.117033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.116979    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117020    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117035    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117080    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117136    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xwj\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-kube-api-access-42xwj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117150    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117166    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117179    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.117397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.117193    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff07ceb-8b79-4ccf-8b28-acce23859fcf-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:53.121088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.121057    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aafd8ff-a536-42ec-9c86-446d2c05a767" path="/var/lib/kubelet/pods/2aafd8ff-a536-42ec-9c86-446d2c05a767/volumes"
Oct 30 03:07:53.597608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.597567    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5" event={"ID":"7ff07ceb-8b79-4ccf-8b28-acce23859fcf","Type":"ContainerDied","Data":"726be06597c913440133c4c5e23c941aca4e6968144a493f39c471196cd86a3a"}
Oct 30 03:07:53.598014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.597631    4214 scope.go:122] "RemoveContainer" containerID="c15024eaa7a394354e0969593a31cee0c757ab49804b2e59b3106a6733b4184c"
Oct 30 03:07:53.598014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.597726    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"
Oct 30 03:07:53.601243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.601209    4214 generic.go:356] "Generic (PLEG): container finished" podID="359e6cf9-4755-4460-b34b-7373de66a212" containerID="e903d457c550f39e346e34049c4545362b6b1ba3c3f05651c0c877bc1fe02837" exitCode=0
Oct 30 03:07:53.601354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.601290    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" event={"ID":"359e6cf9-4755-4460-b34b-7373de66a212","Type":"ContainerDied","Data":"e903d457c550f39e346e34049c4545362b6b1ba3c3f05651c0c877bc1fe02837"}
Oct 30 03:07:53.688284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.688222    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:07:53.694059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.694029    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-k8qd5"]
Oct 30 03:07:53.929282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:53.929245    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:07:54.032419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032383    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032437    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032565    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032614    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032657    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032705    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.032809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032747    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:54.032809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032767    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.033011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032863    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:54.033011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032898    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.033011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032940    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\") pod \"359e6cf9-4755-4460-b34b-7373de66a212\" (UID: \"359e6cf9-4755-4460-b34b-7373de66a212\") "
Oct 30 03:07:54.033011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.032959    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:54.033210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033195    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:54.033498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033473    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:54.033647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033628    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.033731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033653    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.033731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033666    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.033731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033679    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.033731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.033694    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359e6cf9-4755-4460-b34b-7373de66a212-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.043154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.042392    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:54.043154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.042462    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "kube-api-access-tlp4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:54.043154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.042509    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:54.043154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.042574    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo" pod "359e6cf9-4755-4460-b34b-7373de66a212" (UID: "359e6cf9-4755-4460-b34b-7373de66a212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:07:54.140305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.135978    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359e6cf9-4755-4460-b34b-7373de66a212-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.140305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.136015    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlp4j\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-kube-api-access-tlp4j\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.140305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.136031    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359e6cf9-4755-4460-b34b-7373de66a212-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.140305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.136045    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359e6cf9-4755-4460-b34b-7373de66a212-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:07:54.610628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.610600    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"
Oct 30 03:07:54.611107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.610631    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9" event={"ID":"359e6cf9-4755-4460-b34b-7373de66a212","Type":"ContainerDied","Data":"9fa10bc5a21a51df6d76477dfdf097223b95f743b1945470061f34bbbf267886"}
Oct 30 03:07:54.611107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.610691    4214 scope.go:122] "RemoveContainer" containerID="e903d457c550f39e346e34049c4545362b6b1ba3c3f05651c0c877bc1fe02837"
Oct 30 03:07:54.701004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.700970    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:07:54.705991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:54.705965    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-lsqc9"]
Oct 30 03:07:55.070930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:55.070897    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359e6cf9-4755-4460-b34b-7373de66a212" path="/var/lib/kubelet/pods/359e6cf9-4755-4460-b34b-7373de66a212/volumes"
Oct 30 03:07:55.071716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:07:55.071685    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" path="/var/lib/kubelet/pods/7ff07ceb-8b79-4ccf-8b28-acce23859fcf/volumes"
Oct 30 03:08:05.236992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:05.236959    4214 scope.go:122] "RemoveContainer" containerID="0fe12010f6dc6d6597c28cb8562ab28865232f2d27c99f710020e64bceaf7bb4"
Oct 30 03:08:05.250789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:05.250763    4214 scope.go:122] "RemoveContainer" containerID="eebc693c24efd2402929f36a8aa3d65adea567bb4105edc48a88b3b11727c99c"
Oct 30 03:08:07.081648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.081613    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:08:07.083019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.082993    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2aafd8ff-a536-42ec-9c86-446d2c05a767" containerName="payload-processing"
Oct 30 03:08:07.083019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083019    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aafd8ff-a536-42ec-9c86-446d2c05a767" containerName="payload-processing"
Oct 30 03:08:07.083169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083049    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy"
Oct 30 03:08:07.083169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083058    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy"
Oct 30 03:08:07.083169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083091    4214 cpu_manager.go:414] "containerMap: removing container" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy"
Oct 30 03:08:07.083169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083099    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy"
Oct 30 03:08:07.083370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083352    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aafd8ff-a536-42ec-9c86-446d2c05a767" containerName="payload-processing"
Oct 30 03:08:07.083412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083380    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff07ceb-8b79-4ccf-8b28-acce23859fcf" containerName="istio-proxy"
Oct 30 03:08:07.083412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.083394    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="359e6cf9-4755-4460-b34b-7373de66a212" containerName="istio-proxy"
Oct 30 03:08:07.088602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.088585    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.128622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.128595    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:08:07.139451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.139425    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cea203\"/\"kube-root-ca.crt\""
Oct 30 03:08:07.152546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.152522    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cea203\"/\"default-dockercfg-t79ft\""
Oct 30 03:08:07.152663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.152637    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cea203\"/\"test-model-cea203-kserve-self-signed-certs\""
Oct 30 03:08:07.197825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.197794    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.197983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.197961    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.198074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.198061    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfdl\" (UniqueName: \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.198110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.198098    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.198314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.198290    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.198398 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.198377    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.216787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.216765    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cea203\"/\"openshift-service-ca.crt\""
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301347    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301421    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301555    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301632    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301722    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfdl\" (UniqueName: \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.301777    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.303112    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.303790    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.305574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.304055    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.341293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.335548    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.358250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.354879    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfdl\" (UniqueName: \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.358250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.355552    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\") pod \"test-model-cea203-kserve-845dc5785f-2747g\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") " pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:07.406258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.405413    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:08.061426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:07.995120    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:08:08.814848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:08.814801    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" event={"ID":"be83b073-8355-4439-b9bf-a38114aa7484","Type":"ContainerStarted","Data":"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"}
Oct 30 03:08:08.814848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:08.814852    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" event={"ID":"be83b073-8355-4439-b9bf-a38114aa7484","Type":"ContainerStarted","Data":"d0d2a1f54f105742364bedc718db74a7d47da9383c89b4b234174374e7c297eb"}
Oct 30 03:08:08.896197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:08.896145    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" podStartSLOduration=1.896131928 podStartE2EDuration="1.896131928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:08.890518199 +0000 UTC m=+1627.964688989" watchObservedRunningTime="2026-10-30 03:08:08.896131928 +0000 UTC m=+1627.970302721"
Oct 30 03:08:09.175142 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:09.175039    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:08:09.407803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:09.407770    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:17.408553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:17.408519    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:17.422796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:17.422768    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:17.993162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:17.993132    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:08:24.070837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.070803    4214 generic.go:356] "Generic (PLEG): container finished" podID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerID="7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0" exitCode=0
Oct 30 03:08:24.071476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.070875    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerDied","Data":"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"}
Oct 30 03:08:24.071544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.071488    4214 scope.go:122] "RemoveContainer" containerID="7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"
Oct 30 03:08:24.074193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.074153    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:24.074357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.074302    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:24.074411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.074356    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:24.232981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.232944    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:08:24.242663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.242634    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.294249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.282540    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-67440d-serving-cert\""
Oct 30 03:08:24.294476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.291300    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:08:24.422052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.421933    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.422293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.422243    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgsn\" (UniqueName: \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.422426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.422300    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.422515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.422437    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.532305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.528573    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.532305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.529522    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.532305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.529612    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.532305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.529785    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgsn\" (UniqueName: \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.532305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.529813    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.560711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.560678    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.566882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.566853    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.576204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.576169    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgsn\" (UniqueName: \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\") pod \"maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:24.863382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:24.863341    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:25.087678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:25.087642    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerStarted","Data":"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"}
Oct 30 03:08:25.228563 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:08:25.228526    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab12aee_96be_4959_990d_8233cd65da07.slice/crio-dbdae0e64589196cdd4beead6c9356726be06ea5928ba3d4021eb7608f97f0eb WatchSource:0}: Error finding container dbdae0e64589196cdd4beead6c9356726be06ea5928ba3d4021eb7608f97f0eb: Status 404 returned error can't find the container with id dbdae0e64589196cdd4beead6c9356726be06ea5928ba3d4021eb7608f97f0eb
Oct 30 03:08:25.229747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:25.229723    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:08:25.829763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:25.829739    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:08:25.830940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:25.830917    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:25.831033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:25.830970    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:26.162989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.162941    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" event={"ID":"0ab12aee-96be-4959-990d-8233cd65da07","Type":"ContainerStarted","Data":"c82f1e1bdf3d45f473d877ab4ce7c8c5c711c45de434ec45e21ffd931571c493"}
Oct 30 03:08:26.163854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.163828    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" event={"ID":"0ab12aee-96be-4959-990d-8233cd65da07","Type":"ContainerStarted","Data":"dbdae0e64589196cdd4beead6c9356726be06ea5928ba3d4021eb7608f97f0eb"}
Oct 30 03:08:26.164042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.164030    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:26.289906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.289840    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" podStartSLOduration=1.763961028 podStartE2EDuration="2.289823695s" totalImagesPullingTime="525.862667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:26.256609113 +0000 UTC m=+1645.330779907" watchObservedRunningTime="2026-10-30 03:08:26.289823695 +0000 UTC m=+1645.363994487"
Oct 30 03:08:26.829588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.829551    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:26.829775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:26.829604    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:27.829512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:27.829477    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:27.829981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:27.829530    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:28.830166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:28.830132    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:28.830560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:28.830198    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:28.885955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:28.885050    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:08:29.613840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:29.613808    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:08:29.677328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:29.677293    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:08:29.829740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:29.829711    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:29.829919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:29.829760    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:30.830181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:30.830139    4214 patch_prober.go:29] interesting pod/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:08:30.830616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:30.830192    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:08:31.840252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:31.835771    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:08:31.840252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:31.836983    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:08:33.194715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:33.194689    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:08:34.614250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:34.614183    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" podUID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" containerName="payload-processing" containerID="cri-o://e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c" gracePeriod=25
Oct 30 03:08:34.678367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:34.678323    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" podUID="6a9a0261-e710-484a-a436-2a7befa7190a" containerName="payload-pre-processing" containerID="cri-o://70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f" gracePeriod=25
Oct 30 03:08:35.042602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.042573    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:08:35.107621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.107597    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:08:35.150122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.150030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\") pod \"6a9a0261-e710-484a-a436-2a7befa7190a\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") "
Oct 30 03:08:35.150122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.150093    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\") pod \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") "
Oct 30 03:08:35.150429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.150293    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\" (UniqueName: \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\") pod \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\" (UID: \"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23\") "
Oct 30 03:08:35.150429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.150387    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\" (UniqueName: \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\") pod \"6a9a0261-e710-484a-a436-2a7befa7190a\" (UID: \"6a9a0261-e710-484a-a436-2a7befa7190a\") "
Oct 30 03:08:35.150787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.150761    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume" pod "322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" (UID: "322dbd26-9ec2-4cf2-8db1-ffa32bdacb23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:35.151142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.151118    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:35.152264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.152202    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume" pod "6a9a0261-e710-484a-a436-2a7befa7190a" (UID: "6a9a0261-e710-484a-a436-2a7befa7190a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:35.153653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.153623    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq" pod "6a9a0261-e710-484a-a436-2a7befa7190a" (UID: "6a9a0261-e710-484a-a436-2a7befa7190a"). InnerVolumeSpecName "kube-api-access-94vpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:35.155076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.155044    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6" pod "322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" (UID: "322dbd26-9ec2-4cf2-8db1-ffa32bdacb23"). InnerVolumeSpecName "kube-api-access-n6rh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:35.252311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.252278    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vpq\" (UniqueName: \"kubernetes.io/projected/6a9a0261-e710-484a-a436-2a7befa7190a-kube-api-access-94vpq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:35.252311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.252311    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9a0261-e710-484a-a436-2a7befa7190a-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:35.252311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.252329    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6rh6\" (UniqueName: \"kubernetes.io/projected/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23-kube-api-access-n6rh6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:35.261193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.261160    4214 generic.go:356] "Generic (PLEG): container finished" podID="6a9a0261-e710-484a-a436-2a7befa7190a" containerID="70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f" exitCode=0
Oct 30 03:08:35.261374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.261208    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"
Oct 30 03:08:35.261374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.261275    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" event={"ID":"6a9a0261-e710-484a-a436-2a7befa7190a","Type":"ContainerDied","Data":"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"}
Oct 30 03:08:35.261374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.261330    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9" event={"ID":"6a9a0261-e710-484a-a436-2a7befa7190a","Type":"ContainerDied","Data":"1e8efdd5fa698e8567a06603cb7831b7fdd50605f46fe53093fe154271438bde"}
Oct 30 03:08:35.261374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.261361    4214 scope.go:122] "RemoveContainer" containerID="70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"
Oct 30 03:08:35.263275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.263248    4214 generic.go:356] "Generic (PLEG): container finished" podID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" containerID="e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c" exitCode=0
Oct 30 03:08:35.263372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.263321    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" event={"ID":"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23","Type":"ContainerDied","Data":"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"}
Oct 30 03:08:35.263372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.263362    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr" event={"ID":"322dbd26-9ec2-4cf2-8db1-ffa32bdacb23","Type":"ContainerDied","Data":"1649dcb7d898728a1d02e611656e5fc8ba34e488efdc066494af4b4e6df46395"}
Oct 30 03:08:35.263517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.263499    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"
Oct 30 03:08:35.277842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.277814    4214 scope.go:122] "RemoveContainer" containerID="70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"
Oct 30 03:08:35.278191 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:35.278168    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f\": container with ID starting with 70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f not found: ID does not exist" containerID="70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"
Oct 30 03:08:35.278320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.278202    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f"} err="failed to get container status \"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f\": rpc error: code = NotFound desc = could not find container \"70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f\": container with ID starting with 70ab514fcdc2fe63acf478391d1abf7cb15b34395aae67b50200e658514a039f not found: ID does not exist"
Oct 30 03:08:35.278320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.278225    4214 scope.go:122] "RemoveContainer" containerID="e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"
Oct 30 03:08:35.290785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.290764    4214 scope.go:122] "RemoveContainer" containerID="e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"
Oct 30 03:08:35.291117 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:35.291094    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c\": container with ID starting with e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c not found: ID does not exist" containerID="e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"
Oct 30 03:08:35.291197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.291124    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c"} err="failed to get container status \"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c\": rpc error: code = NotFound desc = could not find container \"e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c\": container with ID starting with e4206cdc995f0460aa6ba4278d0a1afda5542c3576fa82a9abf3efb6d733fa7c not found: ID does not exist"
Oct 30 03:08:35.319912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.319888    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:08:35.325187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.325164    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-w7dn9"]
Oct 30 03:08:35.363513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.363444    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:08:35.371304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:35.371258    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-v9fcr"]
Oct 30 03:08:37.071217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:37.071181    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" path="/var/lib/kubelet/pods/322dbd26-9ec2-4cf2-8db1-ffa32bdacb23/volumes"
Oct 30 03:08:37.071881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:37.071862    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9a0261-e710-484a-a436-2a7befa7190a" path="/var/lib/kubelet/pods/6a9a0261-e710-484a-a436-2a7befa7190a/volumes"
Oct 30 03:08:39.362859 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:39.355814    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:08:39.702579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.702466    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:08:39.703461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703439    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6a9a0261-e710-484a-a436-2a7befa7190a" containerName="payload-pre-processing"
Oct 30 03:08:39.703461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703458    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9a0261-e710-484a-a436-2a7befa7190a" containerName="payload-pre-processing"
Oct 30 03:08:39.703673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703523    4214 cpu_manager.go:414] "containerMap: removing container" podUID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" containerName="payload-processing"
Oct 30 03:08:39.703673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703529    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" containerName="payload-processing"
Oct 30 03:08:39.703673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703653    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9a0261-e710-484a-a436-2a7befa7190a" containerName="payload-pre-processing"
Oct 30 03:08:39.703673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.703675    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="322dbd26-9ec2-4cf2-8db1-ffa32bdacb23" containerName="payload-processing"
Oct 30 03:08:39.707724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.707704    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.729358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.729334    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:08:39.792294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792212    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792289    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792404    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792435    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vr2\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792463    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792586    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792623    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792680    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.792867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.792755    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.810307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.810271    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:08:39.816458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.816433    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.819340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.819309    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:08:39.894327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894285    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.894522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894439    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.894522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894500    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894524    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894610    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894636    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vr2\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894658    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894698    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.894916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894758    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.894916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894806    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.894916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894861    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.894916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894892    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894929    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.894959    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895041    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895068    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fkx\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895099    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.895196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895162    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.895556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895275    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:39.895735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895710    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.895803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.895722    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.896028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.896003    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.896573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.896547    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.897450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.897429    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.898356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.898337    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:39.957317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:39.956193    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vr2\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:40.001811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.001773    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.001854    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.001977    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002003    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fkx\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002029    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002125    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002210    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.002360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002292    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.002767    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.003301    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.003498    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.003605    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.003679    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.004199    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.008456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.007276    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.019244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.018546    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.019244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.018824    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.023328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.023305    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fkx\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.023799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.023777    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.028004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.027846    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:40.136403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.136240    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:40.362026 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:08:40.361996    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3117b6d5_2d5b_40e7_b6a6_53cd00d8fb40.slice/crio-e87db7db90d8a2ae349a41750b37442d6266a627551464fbfe98fecb2b8c5237 WatchSource:0}: Error finding container e87db7db90d8a2ae349a41750b37442d6266a627551464fbfe98fecb2b8c5237: Status 404 returned error can't find the container with id e87db7db90d8a2ae349a41750b37442d6266a627551464fbfe98fecb2b8c5237
Oct 30 03:08:40.363669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.363647    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:08:40.365447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.365417    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.365518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.365478    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.365518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.365507    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.390334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.390296    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" event={"ID":"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40","Type":"ContainerStarted","Data":"e87db7db90d8a2ae349a41750b37442d6266a627551464fbfe98fecb2b8c5237"}
Oct 30 03:08:40.464102 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:08:40.464004    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded907564_a568_401b_b9fc_df7dddf72873.slice/crio-941838f85c786957afe411942f462bbe2e5ec4ddd59098e976b7519ed9324246 WatchSource:0}: Error finding container 941838f85c786957afe411942f462bbe2e5ec4ddd59098e976b7519ed9324246: Status 404 returned error can't find the container with id 941838f85c786957afe411942f462bbe2e5ec4ddd59098e976b7519ed9324246
Oct 30 03:08:40.467072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.467042    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:08:40.467377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.467331    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.467496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.467428    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.467496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.467458    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:40.842457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.842417    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:08:40.877911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:40.877855    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:08:41.230360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.230193    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:08:41.230542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.230449    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" podUID="625bfcdd-107b-496c-bda0-f64657b9e2cf" containerName="maas-api" containerID="cri-o://2e477c2e28c76a333ebb2b05756177f3bdced4cb75b2c7bfcd3bb01293183db5" gracePeriod=30
Oct 30 03:08:41.423792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.423753    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" event={"ID":"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40","Type":"ContainerStarted","Data":"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"}
Oct 30 03:08:41.428393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.428337    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" event={"ID":"ed907564-a568-401b-b9fc-df7dddf72873","Type":"ContainerStarted","Data":"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"}
Oct 30 03:08:41.428393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.428384    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" event={"ID":"ed907564-a568-401b-b9fc-df7dddf72873","Type":"ContainerStarted","Data":"941838f85c786957afe411942f462bbe2e5ec4ddd59098e976b7519ed9324246"}
Oct 30 03:08:41.431955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.431931    4214 generic.go:356] "Generic (PLEG): container finished" podID="625bfcdd-107b-496c-bda0-f64657b9e2cf" containerID="2e477c2e28c76a333ebb2b05756177f3bdced4cb75b2c7bfcd3bb01293183db5" exitCode=0
Oct 30 03:08:41.432157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.431993    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" event={"ID":"625bfcdd-107b-496c-bda0-f64657b9e2cf","Type":"ContainerDied","Data":"2e477c2e28c76a333ebb2b05756177f3bdced4cb75b2c7bfcd3bb01293183db5"}
Oct 30 03:08:41.502844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.502704    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podStartSLOduration=2.502680028 podStartE2EDuration="2.502680028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:41.481737142 +0000 UTC m=+1660.555907945" watchObservedRunningTime="2026-10-30 03:08:41.502680028 +0000 UTC m=+1660.576850823"
Oct 30 03:08:41.566287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:41.534392    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podStartSLOduration=2.534371555 podStartE2EDuration="2.534371555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:41.531459851 +0000 UTC m=+1660.605630649" watchObservedRunningTime="2026-10-30 03:08:41.534371555 +0000 UTC m=+1660.608542348"
Oct 30 03:08:42.056254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.037320    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:42.124314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.101255    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:42.124314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.101328    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:42.196473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.168875    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:42.617276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.609807    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:42.617276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.609871    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:42.800903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.800864    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:08:42.822577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.822532    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr" event={"ID":"625bfcdd-107b-496c-bda0-f64657b9e2cf","Type":"ContainerDied","Data":"355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff"}
Oct 30 03:08:42.822908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.822884    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355f6e02c717c07a7dcdf1cbbb37391be8d7d1ebce7273243148cd32c772f5ff"
Oct 30 03:08:42.824411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.824389    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:08:42.914872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.914833    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\") pod \"625bfcdd-107b-496c-bda0-f64657b9e2cf\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") "
Oct 30 03:08:42.915057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.915045    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\" (UniqueName: \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\") pod \"625bfcdd-107b-496c-bda0-f64657b9e2cf\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") "
Oct 30 03:08:42.915205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.915142    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\") pod \"625bfcdd-107b-496c-bda0-f64657b9e2cf\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") "
Oct 30 03:08:42.915340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.915257    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\") pod \"625bfcdd-107b-496c-bda0-f64657b9e2cf\" (UID: \"625bfcdd-107b-496c-bda0-f64657b9e2cf\") "
Oct 30 03:08:42.917795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.917727    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca" pod "625bfcdd-107b-496c-bda0-f64657b9e2cf" (UID: "625bfcdd-107b-496c-bda0-f64657b9e2cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:42.917914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.917777    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls" pod "625bfcdd-107b-496c-bda0-f64657b9e2cf" (UID: "625bfcdd-107b-496c-bda0-f64657b9e2cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:42.917914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.917855    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6" pod "625bfcdd-107b-496c-bda0-f64657b9e2cf" (UID: "625bfcdd-107b-496c-bda0-f64657b9e2cf"). InnerVolumeSpecName "kube-api-access-rb8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:42.920688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:42.920657    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs" pod "625bfcdd-107b-496c-bda0-f64657b9e2cf" (UID: "625bfcdd-107b-496c-bda0-f64657b9e2cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:43.018312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.018262    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:43.018312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.018297    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625bfcdd-107b-496c-bda0-f64657b9e2cf-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:43.018312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.018313    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625bfcdd-107b-496c-bda0-f64657b9e2cf-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:43.018636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.018329    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8f6\" (UniqueName: \"kubernetes.io/projected/625bfcdd-107b-496c-bda0-f64657b9e2cf-kube-api-access-rb8f6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:43.028132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.028096    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:43.028286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.028145    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:43.147351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.140623    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:43.147351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.140683    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:43.830007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.829919    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"
Oct 30 03:08:43.900513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.900476    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:08:43.912700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:43.912667    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-44a8fd-74cc4cf5c9-sbzdr"]
Oct 30 03:08:44.028518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:44.028484    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:44.028694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:44.028533    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:44.139317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:44.139051    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:44.139317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:44.139108    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:45.032582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.032545    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:45.033167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.033136    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:45.100310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.095111    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625bfcdd-107b-496c-bda0-f64657b9e2cf" path="/var/lib/kubelet/pods/625bfcdd-107b-496c-bda0-f64657b9e2cf/volumes"
Oct 30 03:08:45.156255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.148434    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:45.156255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.148492    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:45.846343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.846291    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" podUID="57908fae-37b2-4542-9d96-aa109aba4bb0" containerName="payload-processing" containerID="cri-o://5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7" gracePeriod=25
Oct 30 03:08:45.895266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:45.895208    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" podUID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" containerName="payload-pre-processing" containerID="cri-o://2f16efc484070e0fa141e1c7780f3ec566c9dd731c5f96a30f8a39949d8c9061" gracePeriod=25
Oct 30 03:08:46.029197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.029146    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:46.029333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.029203    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:46.158309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.157276    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:46.158309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.157358    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:46.206837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.206752    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:08:46.209273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.208065    4214 cpu_manager.go:414] "containerMap: removing container" podUID="625bfcdd-107b-496c-bda0-f64657b9e2cf" containerName="maas-api"
Oct 30 03:08:46.209273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.208091    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625bfcdd-107b-496c-bda0-f64657b9e2cf" containerName="maas-api"
Oct 30 03:08:46.209273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.208419    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="625bfcdd-107b-496c-bda0-f64657b9e2cf" containerName="maas-api"
Oct 30 03:08:46.214439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.213285    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.223483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.221399    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:08:46.230668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.230631    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-16df10-openshift-default-dockercfg-dpznz\""
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.446821    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.446928    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.446969    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447030    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447174    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447274    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447299    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447410    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6vh\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.478321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.447446    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652334    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652418    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652446    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652500    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652613    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652682    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652705    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652804    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6vh\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.652840    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.653517    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.675294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.654129    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.755635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.700627    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.755635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.701223    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.755635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.701508    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.796489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.793901    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.854438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.851963    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.854438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.853065    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.926486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.923882    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6vh\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\") pod \"e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") " pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:46.993704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.972676    4214 generic.go:356] "Generic (PLEG): container finished" podID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" containerID="2f16efc484070e0fa141e1c7780f3ec566c9dd731c5f96a30f8a39949d8c9061" exitCode=0
Oct 30 03:08:46.993704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.972839    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" event={"ID":"cb40834a-d7e8-4f20-86ac-0d28a2f81664","Type":"ContainerDied","Data":"2f16efc484070e0fa141e1c7780f3ec566c9dd731c5f96a30f8a39949d8c9061"}
Oct 30 03:08:46.993704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.989338    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:08:47.002443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.997676    4214 generic.go:356] "Generic (PLEG): container finished" podID="57908fae-37b2-4542-9d96-aa109aba4bb0" containerID="5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7" exitCode=0
Oct 30 03:08:47.002443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.997867    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" event={"ID":"57908fae-37b2-4542-9d96-aa109aba4bb0","Type":"ContainerDied","Data":"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"}
Oct 30 03:08:47.002443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.997916    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr" event={"ID":"57908fae-37b2-4542-9d96-aa109aba4bb0","Type":"ContainerDied","Data":"709d3856279f9afca9790edb2b0a5068298d3a7f141038b63fb244628431a6db"}
Oct 30 03:08:47.002443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:46.997944    4214 scope.go:122] "RemoveContainer" containerID="5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"
Oct 30 03:08:47.020337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.020310    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:47.032359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.032188    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 03:08:47.032601 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.032562    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 03:08:47.116699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.116674    4214 scope.go:122] "RemoveContainer" containerID="5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"
Oct 30 03:08:47.120963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.120930    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\" (UniqueName: \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\") pod \"57908fae-37b2-4542-9d96-aa109aba4bb0\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") "
Oct 30 03:08:47.121246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.121214    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\") pod \"57908fae-37b2-4542-9d96-aa109aba4bb0\" (UID: \"57908fae-37b2-4542-9d96-aa109aba4bb0\") "
Oct 30 03:08:47.122810 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:47.122787    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7\": container with ID starting with 5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7 not found: ID does not exist" containerID="5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"
Oct 30 03:08:47.123165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.123128    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7"} err="failed to get container status \"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7\": rpc error: code = NotFound desc = could not find container \"5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7\": container with ID starting with 5e00ce409effd442c0eb52fffb7c3fc2f6e04890b39f59caaef7e8c1bab3c4b7 not found: ID does not exist"
Oct 30 03:08:47.123319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.123094    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume" pod "57908fae-37b2-4542-9d96-aa109aba4bb0" (UID: "57908fae-37b2-4542-9d96-aa109aba4bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:47.126727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.126662    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc" pod "57908fae-37b2-4542-9d96-aa109aba4bb0" (UID: "57908fae-37b2-4542-9d96-aa109aba4bb0"). InnerVolumeSpecName "kube-api-access-f4rwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:47.141293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.137574    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:47.141293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.137623    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:47.224801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.224136    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57908fae-37b2-4542-9d96-aa109aba4bb0-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:47.224801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.224168    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rwc\" (UniqueName: \"kubernetes.io/projected/57908fae-37b2-4542-9d96-aa109aba4bb0-kube-api-access-f4rwc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:47.297528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.295460    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:08:47.430704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.429248    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\" (UniqueName: \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\") pod \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") "
Oct 30 03:08:47.430704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.429371    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\") pod \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\" (UID: \"cb40834a-d7e8-4f20-86ac-0d28a2f81664\") "
Oct 30 03:08:47.465252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.434432    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume" pod "cb40834a-d7e8-4f20-86ac-0d28a2f81664" (UID: "cb40834a-d7e8-4f20-86ac-0d28a2f81664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:47.594413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.539848    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb40834a-d7e8-4f20-86ac-0d28a2f81664-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:47.594413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.559660    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw" pod "cb40834a-d7e8-4f20-86ac-0d28a2f81664" (UID: "cb40834a-d7e8-4f20-86ac-0d28a2f81664"). InnerVolumeSpecName "kube-api-access-sp6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:47.649892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.649852    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp6nw\" (UniqueName: \"kubernetes.io/projected/cb40834a-d7e8-4f20-86ac-0d28a2f81664-kube-api-access-sp6nw\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:47.662315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.656850    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:08:47.663107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.663066    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:47.663209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.663141    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:47.663209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:47.663177    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:48.014957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.014856    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"
Oct 30 03:08:48.022864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.022565    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerStarted","Data":"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"}
Oct 30 03:08:48.022864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.022628    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerStarted","Data":"aeee982586d97241d5845974bf6e1e6811f93b7d69ec9330818c1e083f98d138"}
Oct 30 03:08:48.024757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.024727    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc" event={"ID":"cb40834a-d7e8-4f20-86ac-0d28a2f81664","Type":"ContainerDied","Data":"29436c77ce10ab4349afda3d2051b96f6638f00cff8a672fe771660415e7d3b1"}
Oct 30 03:08:48.024901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.024788    4214 scope.go:122] "RemoveContainer" containerID="2f16efc484070e0fa141e1c7780f3ec566c9dd731c5f96a30f8a39949d8c9061"
Oct 30 03:08:48.024901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.024862    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"
Oct 30 03:08:48.092002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.091969    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:08:48.107001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.106970    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3337a-cfbc4f494-z9pkr"]
Oct 30 03:08:48.138196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.137323    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 03:08:48.138196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.137378    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 03:08:48.147878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.147819    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podStartSLOduration=2.14780052 podStartE2EDuration="2.14780052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:48.141803372 +0000 UTC m=+1667.215974166" watchObservedRunningTime="2026-10-30 03:08:48.14780052 +0000 UTC m=+1667.221971314"
Oct 30 03:08:48.209369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.209334    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:08:48.216964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.216936    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3337a-679664c6d7-gb2jc"]
Oct 30 03:08:48.444910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.444874    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:48.445768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.445745    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:48.446760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:48.446733    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:08:49.021107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.021079    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:49.027199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.027177    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:49.033844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.033821    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:49.034416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.034394    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:08:49.078812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.078773    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57908fae-37b2-4542-9d96-aa109aba4bb0" path="/var/lib/kubelet/pods/57908fae-37b2-4542-9d96-aa109aba4bb0/volumes"
Oct 30 03:08:49.079663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.079635    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" path="/var/lib/kubelet/pods/cb40834a-d7e8-4f20-86ac-0d28a2f81664/volumes"
Oct 30 03:08:49.143211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.143180    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:49.143669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.143646    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:49.144465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:49.144444    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:08:52.691988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.691955    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:08:52.692884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.692865    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" containerName="payload-pre-processing"
Oct 30 03:08:52.692942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.692886    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" containerName="payload-pre-processing"
Oct 30 03:08:52.692942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.692926    4214 cpu_manager.go:414] "containerMap: removing container" podUID="57908fae-37b2-4542-9d96-aa109aba4bb0" containerName="payload-processing"
Oct 30 03:08:52.692942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.692931    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57908fae-37b2-4542-9d96-aa109aba4bb0" containerName="payload-processing"
Oct 30 03:08:52.693100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.693060    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb40834a-d7e8-4f20-86ac-0d28a2f81664" containerName="payload-pre-processing"
Oct 30 03:08:52.693100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.693075    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="57908fae-37b2-4542-9d96-aa109aba4bb0" containerName="payload-processing"
Oct 30 03:08:52.696630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.696611    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:52.723173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.723143    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:08:52.734142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.729514    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-16df10-dockercfg-b82rj\""
Oct 30 03:08:52.734142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.729610    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-16df10\""
Oct 30 03:08:52.832686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.832639    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:52.832883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.832759    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8qz\" (UniqueName: \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:52.915339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.911737    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:08:52.947470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.938500    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:52.947470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.938618    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8qz\" (UniqueName: \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:52.947470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:52.947365    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:53.121790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.121757    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8qz\" (UniqueName: \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\") pod \"payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:53.137465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.137433    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.222400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.222368    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:08:53.235428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.235398    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5cqd\" (UniqueName: \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.235724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.235698    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.312697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.312662    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:53.340346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.338435    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5cqd\" (UniqueName: \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.340346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.338965    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.340346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.339917    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.363483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.363176    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5cqd\" (UniqueName: \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\") pod \"payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.514879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.514800    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:53.717089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.717041    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:08:53.727561 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:08:53.727523    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd00b2bf_87ec_44e2_9696_74fff694ab8c.slice/crio-70ce291b858c82e0c8adfacb88c73075882281b111e53c178a82a799c77dd964 WatchSource:0}: Error finding container 70ce291b858c82e0c8adfacb88c73075882281b111e53c178a82a799c77dd964: Status 404 returned error can't find the container with id 70ce291b858c82e0c8adfacb88c73075882281b111e53c178a82a799c77dd964
Oct 30 03:08:53.840857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:53.840754    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:08:54.211348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.211309    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" event={"ID":"bd00b2bf-87ec-44e2-9696-74fff694ab8c","Type":"ContainerStarted","Data":"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"}
Oct 30 03:08:54.211536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.211366    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" event={"ID":"bd00b2bf-87ec-44e2-9696-74fff694ab8c","Type":"ContainerStarted","Data":"70ce291b858c82e0c8adfacb88c73075882281b111e53c178a82a799c77dd964"}
Oct 30 03:08:54.211536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.211411    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:08:54.213222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.213199    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" event={"ID":"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5","Type":"ContainerStarted","Data":"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"}
Oct 30 03:08:54.213342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.213275    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:08:54.213342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.213293    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" event={"ID":"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5","Type":"ContainerStarted","Data":"f1af7ba1ce4573e34f153f8649e653b4e98fb682ed1693b8d8c5e83971734a01"}
Oct 30 03:08:54.256853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.256799    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" podStartSLOduration=2.25678293 podStartE2EDuration="2.25678293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:54.256611791 +0000 UTC m=+1673.330782585" watchObservedRunningTime="2026-10-30 03:08:54.25678293 +0000 UTC m=+1673.330953724"
Oct 30 03:08:54.298738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:54.298682    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" podStartSLOduration=2.298668752 podStartE2EDuration="2.298668752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:54.296613916 +0000 UTC m=+1673.370784708" watchObservedRunningTime="2026-10-30 03:08:54.298668752 +0000 UTC m=+1673.372839545"
Oct 30 03:08:55.126431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:55.124274    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:08:55.126431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:55.124538    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" podUID="095850da-0b02-4df3-b98c-a3c450e5eac3" containerName="maas-api" containerID="cri-o://e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43" gracePeriod=30
Oct 30 03:08:55.978464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:55.976505    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:08:56.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.022199    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\" (UniqueName: \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\") pod \"095850da-0b02-4df3-b98c-a3c450e5eac3\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") "
Oct 30 03:08:56.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.022324    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\") pod \"095850da-0b02-4df3-b98c-a3c450e5eac3\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") "
Oct 30 03:08:56.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.022390    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\") pod \"095850da-0b02-4df3-b98c-a3c450e5eac3\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") "
Oct 30 03:08:56.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.022673    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\") pod \"095850da-0b02-4df3-b98c-a3c450e5eac3\" (UID: \"095850da-0b02-4df3-b98c-a3c450e5eac3\") "
Oct 30 03:08:56.043262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.039609    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs" pod "095850da-0b02-4df3-b98c-a3c450e5eac3" (UID: "095850da-0b02-4df3-b98c-a3c450e5eac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:56.065274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.063413    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4" pod "095850da-0b02-4df3-b98c-a3c450e5eac3" (UID: "095850da-0b02-4df3-b98c-a3c450e5eac3"). InnerVolumeSpecName "kube-api-access-h8np4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:56.069254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.066331    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca" pod "095850da-0b02-4df3-b98c-a3c450e5eac3" (UID: "095850da-0b02-4df3-b98c-a3c450e5eac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:56.085280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.075548    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls" pod "095850da-0b02-4df3-b98c-a3c450e5eac3" (UID: "095850da-0b02-4df3-b98c-a3c450e5eac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:56.125012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.124978    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095850da-0b02-4df3-b98c-a3c450e5eac3-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.125268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.125209    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8np4\" (UniqueName: \"kubernetes.io/projected/095850da-0b02-4df3-b98c-a3c450e5eac3-kube-api-access-h8np4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.125422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.125403    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.125524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.125512    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095850da-0b02-4df3-b98c-a3c450e5eac3-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.245259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.242691    4214 generic.go:356] "Generic (PLEG): container finished" podID="095850da-0b02-4df3-b98c-a3c450e5eac3" containerID="e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43" exitCode=0
Oct 30 03:08:56.245259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.243089    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" event={"ID":"095850da-0b02-4df3-b98c-a3c450e5eac3","Type":"ContainerDied","Data":"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"}
Oct 30 03:08:56.245259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.243201    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b" event={"ID":"095850da-0b02-4df3-b98c-a3c450e5eac3","Type":"ContainerDied","Data":"56eee315b316a868af962d5203e11b6be97127357cf17e1acfaef26360772429"}
Oct 30 03:08:56.245259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.243370    4214 scope.go:122] "RemoveContainer" containerID="e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"
Oct 30 03:08:56.245259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.243672    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"
Oct 30 03:08:56.412762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.409223    4214 scope.go:122] "RemoveContainer" containerID="e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"
Oct 30 03:08:56.412762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.409454    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:08:56.412762 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:08:56.409757    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43\": container with ID starting with e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43 not found: ID does not exist" containerID="e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"
Oct 30 03:08:56.412762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.409799    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43"} err="failed to get container status \"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43\": rpc error: code = NotFound desc = could not find container \"e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43\": container with ID starting with e4284689aaf3a03419b764aad4e77a15ca19157f4e0766e3a64c0d69f81d4a43 not found: ID does not exist"
Oct 30 03:08:56.420453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:56.420112    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-be9f2a59-697585b688-dlb6b"]
Oct 30 03:08:57.072958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:08:57.072921    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095850da-0b02-4df3-b98c-a3c450e5eac3" path="/var/lib/kubelet/pods/095850da-0b02-4df3-b98c-a3c450e5eac3/volumes"
Oct 30 03:09:01.853609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:01.853563    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:09:01.887260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:01.887189    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:09:05.009069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.003536    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:09:05.009069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.003799    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerName="maas-api" containerID="cri-o://ea57aaf3b9ee751b6e60f6ced996e63f5839ca1ef50b3ec9582950e6a1915da0" gracePeriod=30
Oct 30 03:09:05.225096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.225062    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:09:05.225730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.225707    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:09:05.876295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.876256    4214 generic.go:356] "Generic (PLEG): container finished" podID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerID="ea57aaf3b9ee751b6e60f6ced996e63f5839ca1ef50b3ec9582950e6a1915da0" exitCode=0
Oct 30 03:09:05.876541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.876518    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" event={"ID":"c5b4399a-07e2-45f1-a2c0-308b2765a636","Type":"ContainerDied","Data":"ea57aaf3b9ee751b6e60f6ced996e63f5839ca1ef50b3ec9582950e6a1915da0"}
Oct 30 03:09:05.898626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:05.898560    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:09:06.011494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.011461    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\") pod \"c5b4399a-07e2-45f1-a2c0-308b2765a636\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") "
Oct 30 03:09:06.011494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.011502    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\") pod \"c5b4399a-07e2-45f1-a2c0-308b2765a636\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") "
Oct 30 03:09:06.012041 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.011723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\") pod \"c5b4399a-07e2-45f1-a2c0-308b2765a636\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") "
Oct 30 03:09:06.012041 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.011796    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\" (UniqueName: \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\") pod \"c5b4399a-07e2-45f1-a2c0-308b2765a636\" (UID: \"c5b4399a-07e2-45f1-a2c0-308b2765a636\") "
Oct 30 03:09:06.012159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.012130    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca" pod "c5b4399a-07e2-45f1-a2c0-308b2765a636" (UID: "c5b4399a-07e2-45f1-a2c0-308b2765a636"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:06.012735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.012707    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b4399a-07e2-45f1-a2c0-308b2765a636-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:06.015325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.015299    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls" pod "c5b4399a-07e2-45f1-a2c0-308b2765a636" (UID: "c5b4399a-07e2-45f1-a2c0-308b2765a636"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:06.015430 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.015355    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p" pod "c5b4399a-07e2-45f1-a2c0-308b2765a636" (UID: "c5b4399a-07e2-45f1-a2c0-308b2765a636"). InnerVolumeSpecName "kube-api-access-f4f4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:06.018342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.018321    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs" pod "c5b4399a-07e2-45f1-a2c0-308b2765a636" (UID: "c5b4399a-07e2-45f1-a2c0-308b2765a636"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:06.114793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.114759    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4f4p\" (UniqueName: \"kubernetes.io/projected/c5b4399a-07e2-45f1-a2c0-308b2765a636-kube-api-access-f4f4p\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:06.114793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.114794    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:06.115041 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.114808    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5b4399a-07e2-45f1-a2c0-308b2765a636-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:06.867296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.854306    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" podUID="12702691-000a-49da-8c40-9464c13b953c" containerName="payload-processing" containerID="cri-o://c77bdf3a96e8f8cd1828eb3fbaef98ef30b6c95ff455a6d74ef18d31205e523d" gracePeriod=25
Oct 30 03:09:06.890728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.888865    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" podUID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" containerName="payload-pre-processing" containerID="cri-o://049701df9920ac16df24fe02fa7730478a5074090f8197192558b40e23692b0f" gracePeriod=25
Oct 30 03:09:06.947253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.930309    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" event={"ID":"c5b4399a-07e2-45f1-a2c0-308b2765a636","Type":"ContainerDied","Data":"6b040d48eed4ac37b45433578a13d42e6d4a0f279bc614ac291936a43e8cda79"}
Oct 30 03:09:06.947253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.930380    4214 scope.go:122] "RemoveContainer" containerID="ea57aaf3b9ee751b6e60f6ced996e63f5839ca1ef50b3ec9582950e6a1915da0"
Oct 30 03:09:06.947253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:06.930558    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"
Oct 30 03:09:07.209487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.200382    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:09:07.231284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.211332    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x"]
Oct 30 03:09:07.978248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.975571    4214 generic.go:356] "Generic (PLEG): container finished" podID="12702691-000a-49da-8c40-9464c13b953c" containerID="c77bdf3a96e8f8cd1828eb3fbaef98ef30b6c95ff455a6d74ef18d31205e523d" exitCode=0
Oct 30 03:09:07.978248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.975971    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" event={"ID":"12702691-000a-49da-8c40-9464c13b953c","Type":"ContainerDied","Data":"c77bdf3a96e8f8cd1828eb3fbaef98ef30b6c95ff455a6d74ef18d31205e523d"}
Oct 30 03:09:07.978248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.976034    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j" event={"ID":"12702691-000a-49da-8c40-9464c13b953c","Type":"ContainerDied","Data":"f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85"}
Oct 30 03:09:07.978248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.976059    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f36770ac2ca69365d34317d58bfb72a3630a46cf3639490f6ac60a1015625d85"
Oct 30 03:09:07.978248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.977148    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:09:07.981026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.980595    4214 generic.go:356] "Generic (PLEG): container finished" podID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" containerID="049701df9920ac16df24fe02fa7730478a5074090f8197192558b40e23692b0f" exitCode=0
Oct 30 03:09:07.981026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:07.980672    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" event={"ID":"80c29d85-597b-45f2-a692-b8d8d5c9d9d5","Type":"ContainerDied","Data":"049701df9920ac16df24fe02fa7730478a5074090f8197192558b40e23692b0f"}
Oct 30 03:09:08.126066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.126035    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\") pod \"12702691-000a-49da-8c40-9464c13b953c\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") "
Oct 30 03:09:08.126505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.126477    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume" pod "12702691-000a-49da-8c40-9464c13b953c" (UID: "12702691-000a-49da-8c40-9464c13b953c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:08.126842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.126821    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\" (UniqueName: \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\") pod \"12702691-000a-49da-8c40-9464c13b953c\" (UID: \"12702691-000a-49da-8c40-9464c13b953c\") "
Oct 30 03:09:08.128322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.128185    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12702691-000a-49da-8c40-9464c13b953c-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:08.129706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.129646    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7" pod "12702691-000a-49da-8c40-9464c13b953c" (UID: "12702691-000a-49da-8c40-9464c13b953c"). InnerVolumeSpecName "kube-api-access-wnqs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:08.130647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.130628    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:09:08.229790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.229694    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\") pod \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") "
Oct 30 03:09:08.230267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.229889    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\" (UniqueName: \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\") pod \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\" (UID: \"80c29d85-597b-45f2-a692-b8d8d5c9d9d5\") "
Oct 30 03:09:08.230507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.230479    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnqs7\" (UniqueName: \"kubernetes.io/projected/12702691-000a-49da-8c40-9464c13b953c-kube-api-access-wnqs7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:08.231288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.231257    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume" pod "80c29d85-597b-45f2-a692-b8d8d5c9d9d5" (UID: "80c29d85-597b-45f2-a692-b8d8d5c9d9d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:08.233709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.233024    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf" pod "80c29d85-597b-45f2-a692-b8d8d5c9d9d5" (UID: "80c29d85-597b-45f2-a692-b8d8d5c9d9d5"). InnerVolumeSpecName "kube-api-access-wf2sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:08.288567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.288513    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1abc2-7979fb8f5c-99f5x" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.171:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:09:08.331971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.331924    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:08.331971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:08.331961    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2sf\" (UniqueName: \"kubernetes.io/projected/80c29d85-597b-45f2-a692-b8d8d5c9d9d5-kube-api-access-wf2sf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:09.031298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.030352    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"
Oct 30 03:09:09.031298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.030395    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb" event={"ID":"80c29d85-597b-45f2-a692-b8d8d5c9d9d5","Type":"ContainerDied","Data":"bb66f3787d253f8d69c88c4b818f8727c5c3dc9801f0e6aba8b1f1ea9a9ee7e3"}
Oct 30 03:09:09.031298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.030448    4214 scope.go:122] "RemoveContainer" containerID="049701df9920ac16df24fe02fa7730478a5074090f8197192558b40e23692b0f"
Oct 30 03:09:09.031298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.031119    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"
Oct 30 03:09:09.087446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.087408    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" path="/var/lib/kubelet/pods/c5b4399a-07e2-45f1-a2c0-308b2765a636/volumes"
Oct 30 03:09:09.130293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.130258    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:09:09.176350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.176273    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab09ae-5968dfbd96-5ck5j"]
Oct 30 03:09:09.189840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.189806    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:09:09.201246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:09.201117    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab09ae-789ff776f5-nstcb"]
Oct 30 03:09:09.211384 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:09.211349    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:09:11.114845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:11.114812    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12702691-000a-49da-8c40-9464c13b953c" path="/var/lib/kubelet/pods/12702691-000a-49da-8c40-9464c13b953c/volumes"
Oct 30 03:09:11.115349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:11.115326    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" path="/var/lib/kubelet/pods/80c29d85-597b-45f2-a692-b8d8d5c9d9d5/volumes"
Oct 30 03:09:14.566171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.566136    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:09:14.568091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568063    4214 cpu_manager.go:414] "containerMap: removing container" podUID="12702691-000a-49da-8c40-9464c13b953c" containerName="payload-processing"
Oct 30 03:09:14.568289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568265    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12702691-000a-49da-8c40-9464c13b953c" containerName="payload-processing"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568309    4214 cpu_manager.go:414] "containerMap: removing container" podUID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" containerName="payload-pre-processing"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568319    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" containerName="payload-pre-processing"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568332    4214 cpu_manager.go:414] "containerMap: removing container" podUID="095850da-0b02-4df3-b98c-a3c450e5eac3" containerName="maas-api"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568340    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095850da-0b02-4df3-b98c-a3c450e5eac3" containerName="maas-api"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568404    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerName="maas-api"
Oct 30 03:09:14.568419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568413    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerName="maas-api"
Oct 30 03:09:14.568775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568649    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="12702691-000a-49da-8c40-9464c13b953c" containerName="payload-processing"
Oct 30 03:09:14.568775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568668    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c29d85-597b-45f2-a692-b8d8d5c9d9d5" containerName="payload-pre-processing"
Oct 30 03:09:14.568775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568677    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="095850da-0b02-4df3-b98c-a3c450e5eac3" containerName="maas-api"
Oct 30 03:09:14.568775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.568692    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b4399a-07e2-45f1-a2c0-308b2765a636" containerName="maas-api"
Oct 30 03:09:14.575214 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.575182    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.606841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.606802    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1fce3-openshift-default-dockercfg-l84qc\""
Oct 30 03:09:14.619176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.619145    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:09:14.670764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.670726    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.670953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.670775    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.670953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.670807    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.670997    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fml9m\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.671093    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.671162    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.671338    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.671384    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.671526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.671502    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774525    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774591    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774645    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774687    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774731    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774773    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774858    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fml9m\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.774935    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.775057    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.775965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.775642    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.805921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.776221    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.805921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.776502    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.805921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.799481    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.810260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.807034    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.853561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.853450    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.854290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.854260    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fml9m\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.923093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.923029    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:14.999617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:14.999543    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:15.210962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:15.210871    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:15.575123 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:09:15.575079    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c9d962_30dd_4ec5_9c07_0470736111d2.slice/crio-e921ac8deea726465fdbf1ea05634344994b7d7c1735600142ba52b345abcd74 WatchSource:0}: Error finding container e921ac8deea726465fdbf1ea05634344994b7d7c1735600142ba52b345abcd74: Status 404 returned error can't find the container with id e921ac8deea726465fdbf1ea05634344994b7d7c1735600142ba52b345abcd74
Oct 30 03:09:15.575724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:15.575693    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:09:15.578743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:15.578707    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:15.578824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:15.578778    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:15.578824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:15.578807    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:16.162826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:16.162788    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerStarted","Data":"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"}
Oct 30 03:09:16.163016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:16.162849    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerStarted","Data":"e921ac8deea726465fdbf1ea05634344994b7d7c1735600142ba52b345abcd74"}
Oct 30 03:09:16.211210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:16.211176    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:16.219249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:16.219207    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:16.238693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:16.238646    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podStartSLOduration=2.23863405 podStartE2EDuration="2.23863405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:16.222057045 +0000 UTC m=+1695.296227839" watchObservedRunningTime="2026-10-30 03:09:16.23863405 +0000 UTC m=+1695.312804842"
Oct 30 03:09:17.175430 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:17.175389    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:17.176398 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:17.176376    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:09:17.858250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:17.853792    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:09:18.035251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.034452    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:09:18.035251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.034595    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.068418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.067816    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-734d43-openshift-default-dockercfg-fzfh2\""
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155333    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155441    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155492    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155536    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkppk\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155647    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155731    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155836    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.155980    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.156327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.156028    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257334    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257442    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257538    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257625    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257672    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257822    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257920    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257966    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.257993    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkppk\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.259305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.259092    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.262537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.262068    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.265457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.263211    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.265457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.263658    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.265457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.265408    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.276440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.276365    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.277087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.277036    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.282632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.282129    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.284578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.284523    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkppk\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\") pod \"e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") " pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.462492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.462404    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:09:18.462492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.462473    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:09:18.471638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.471610    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:18.819458 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:09:18.819422    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759dd1fd_8bd1_4282_8847_b948d57c4305.slice/crio-faa73bb306f2728741e19cd61bd901cdb30f921c46e1a42dcff00c452577feb5 WatchSource:0}: Error finding container faa73bb306f2728741e19cd61bd901cdb30f921c46e1a42dcff00c452577feb5: Status 404 returned error can't find the container with id faa73bb306f2728741e19cd61bd901cdb30f921c46e1a42dcff00c452577feb5
Oct 30 03:09:18.823034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.823006    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:09:18.823365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.823329    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:18.823433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.823416    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:18.823469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:18.823458    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:19.206514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.206404    4214 generic.go:356] "Generic (PLEG): container finished" podID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerID="dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c" exitCode=0
Oct 30 03:09:19.206514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.206484    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerDied","Data":"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"}
Oct 30 03:09:19.207050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.207024    4214 scope.go:122] "RemoveContainer" containerID="dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"
Oct 30 03:09:19.209589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.209557    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:19.209727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.209668    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:19.209800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.209745    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:09:19.210050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.209900    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerStarted","Data":"700456c4b704c8e1157a78fe402c309806a9feefd8f01e7e72e0f0e24dfea7ee"}
Oct 30 03:09:19.210050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.209959    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerStarted","Data":"faa73bb306f2728741e19cd61bd901cdb30f921c46e1a42dcff00c452577feb5"}
Oct 30 03:09:19.344254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.334424    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podStartSLOduration=2.334405513 podStartE2EDuration="2.334405513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:19.331583848 +0000 UTC m=+1698.405754660" watchObservedRunningTime="2026-10-30 03:09:19.334405513 +0000 UTC m=+1698.408576306"
Oct 30 03:09:19.472469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.472384    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:19.480500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:19.480463    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:20.220683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.220649    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerStarted","Data":"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"}
Oct 30 03:09:20.221069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.221050    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:20.222123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.222097    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:09:20.533660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.533582    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:09:20.565269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.565103    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.568461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.568433    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:09:20.583222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.583200    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c1fce3\""
Oct 30 03:09:20.583478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.583460    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c1fce3-dockercfg-95kb7\""
Oct 30 03:09:20.630970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.630911    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.632613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.632585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97sv\" (UniqueName: \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.797063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.793887    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.797063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.794146    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97sv\" (UniqueName: \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.797063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.795481    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:20.878122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.877851    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:09:20.914301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.906593    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:09:20.922427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:20.922397    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.043312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.038465    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:09:21.066358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.055140    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:21.102629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.098660    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:21.116009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.115975    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwkt\" (UniqueName: \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.116183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.116018    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.128907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.128840    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97sv\" (UniqueName: \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\") pod \"payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:21.185860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.185834    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:21.218453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.218424    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwkt\" (UniqueName: \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.218453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.218456    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.219207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.219180    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.261253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.261205    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwkt\" (UniqueName: \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\") pod \"payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.485378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.485306    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:21.577469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.577277    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:09:21.879394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.878856    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:21.879394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.878909    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:21.883289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:21.883260    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:09:22.277696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.277582    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" event={"ID":"ff44dcf2-81d8-4044-8ea8-a664383f888b","Type":"ContainerStarted","Data":"dff7cfcb2f1ecbd533e103085b49080f5a530ad9ff055b0775c9b1772d3bc5b9"}
Oct 30 03:09:22.277877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.277849    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:22.277943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.277880    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" event={"ID":"ff44dcf2-81d8-4044-8ea8-a664383f888b","Type":"ContainerStarted","Data":"50910e25791f311ebfe6814fa2788ba73fe2ce92460b532edd821e5e1e58f9c6"}
Oct 30 03:09:22.281887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.281853    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" event={"ID":"b9290e3e-cc98-4f16-b348-109b43f92e30","Type":"ContainerStarted","Data":"d5e02e05ac6caa2a02c7381df4496fc26db0772131bd903f9919809fcad7137b"}
Oct 30 03:09:22.282033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.281907    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" event={"ID":"b9290e3e-cc98-4f16-b348-109b43f92e30","Type":"ContainerStarted","Data":"40c062a3640de49af079ea3e3d67fb30f476efeb4cc01ca0fd0c06bc7f58f00a"}
Oct 30 03:09:22.282161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.282145    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:22.325994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.325950    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" podStartSLOduration=2.325937901 podStartE2EDuration="2.325937901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:22.32052653 +0000 UTC m=+1701.394697323" watchObservedRunningTime="2026-10-30 03:09:22.325937901 +0000 UTC m=+1701.400108694"
Oct 30 03:09:22.372112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.372056    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" podStartSLOduration=2.372039763 podStartE2EDuration="2.372039763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:22.369060046 +0000 UTC m=+1701.443230839" watchObservedRunningTime="2026-10-30 03:09:22.372039763 +0000 UTC m=+1701.446210555"
Oct 30 03:09:22.870014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.869525    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:22.870014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:22.869574    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:23.874414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:23.872337    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:23.874414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:23.872402    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:24.010966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.010930    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:09:24.015869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.015847    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.053807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.053768    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-734d43\""
Oct 30 03:09:24.054305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.054278    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-734d43-dockercfg-4mlsq\""
Oct 30 03:09:24.125305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.119566    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.141261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.139414    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsvc\" (UniqueName: \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.185260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.176489    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:09:24.246281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.244408    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.246281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.244474    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsvc\" (UniqueName: \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.246281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.245750    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.287267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.285194    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:09:24.436340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.435281    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsvc\" (UniqueName: \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\") pod \"payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.537081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.534709    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:09:24.537081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.534964    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.567086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.567042    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26r4\" (UniqueName: \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.567290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.567267    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.633593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.633563    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:24.670669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.670637    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.670906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.670883    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26r4\" (UniqueName: \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.672596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.672570    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.697660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.697603    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26r4\" (UniqueName: \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\") pod \"payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.872251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.871548    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:24.872251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.871620    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:24.903836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.903805    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:24.997217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:24.996656    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:09:25.273125 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:09:25.273089    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441fce81_461e_49b5_b46f_f7528bc005cb.slice/crio-01430bcf130bda97abb7bdf28256e2ab5e6c42960a11f57fbbdbac68cf5ca0cb WatchSource:0}: Error finding container 01430bcf130bda97abb7bdf28256e2ab5e6c42960a11f57fbbdbac68cf5ca0cb: Status 404 returned error can't find the container with id 01430bcf130bda97abb7bdf28256e2ab5e6c42960a11f57fbbdbac68cf5ca0cb
Oct 30 03:09:25.275109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.275079    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:09:25.571168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.571132    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" event={"ID":"441fce81-461e-49b5-b46f-f7528bc005cb","Type":"ContainerStarted","Data":"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"}
Oct 30 03:09:25.571358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.571184    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" event={"ID":"441fce81-461e-49b5-b46f-f7528bc005cb","Type":"ContainerStarted","Data":"01430bcf130bda97abb7bdf28256e2ab5e6c42960a11f57fbbdbac68cf5ca0cb"}
Oct 30 03:09:25.572141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.572124    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:25.574868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.574841    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" event={"ID":"5dba0bb9-16dc-4158-aedf-bf4384558c61","Type":"ContainerStarted","Data":"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"}
Oct 30 03:09:25.574989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.574878    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" event={"ID":"5dba0bb9-16dc-4158-aedf-bf4384558c61","Type":"ContainerStarted","Data":"b57a88a0f88f645be46614e109841453ab27b7098677e978476e9a4937bb0386"}
Oct 30 03:09:25.574989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.574942    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:25.617329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.617287    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" podStartSLOduration=1.617274123 podStartE2EDuration="1.617274123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:25.614456231 +0000 UTC m=+1704.688627026" watchObservedRunningTime="2026-10-30 03:09:25.617274123 +0000 UTC m=+1704.691444916"
Oct 30 03:09:25.670895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.668694    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" podStartSLOduration=2.66867425 podStartE2EDuration="2.66867425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:25.657535478 +0000 UTC m=+1704.731706271" watchObservedRunningTime="2026-10-30 03:09:25.66867425 +0000 UTC m=+1704.742845043"
Oct 30 03:09:25.875308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.869565    4214 patch_prober.go:29] interesting pod/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:09:25.875308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:25.869628    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:09:26.872322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:26.872293    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:09:26.872868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:26.872846    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:09:26.873485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:26.873463    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:09:32.549013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.548979    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:09:32.555000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.554973    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.582186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.582153    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:09:32.583674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.583647    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-16df10-serving-cert\""
Oct 30 03:09:32.632198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.632168    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.632388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.632294    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9th\" (UniqueName: \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.632559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.632529    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.632649 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.632635    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.735681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.735638    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.735998    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.736197    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.736366    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9th\" (UniqueName: \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.737626    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:32.737730    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-serving-cert: secret "maas-api-e2e-inf-a-16df10-serving-cert" not found
Oct 30 03:09:32.752263 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:32.737783    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls podName:63bedab7-7e5d-498e-9069-560e0c933704 nodeName:}" failed. No retries permitted until 2026-10-30 03:09:33.237765661 +0000 UTC m=+1712.311936439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls") pod "maas-api-e2e-inf-a-16df10-64f95587df-72bxg" (UID: "63bedab7-7e5d-498e-9069-560e0c933704") : secret "maas-api-e2e-inf-a-16df10-serving-cert" not found
Oct 30 03:09:32.792444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.791675    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:32.794369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:32.794343    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9th\" (UniqueName: \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:33.281414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:33.281370    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:33.283903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:33.283878    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") pod \"maas-api-e2e-inf-a-16df10-64f95587df-72bxg\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:33.298945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:33.298916    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:09:33.479019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:33.478989    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:33.916130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:33.916041    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:09:34.552317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:34.552285    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:09:34.751616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:34.751528    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" event={"ID":"63bedab7-7e5d-498e-9069-560e0c933704","Type":"ContainerStarted","Data":"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"}
Oct 30 03:09:34.751616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:34.751592    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" event={"ID":"63bedab7-7e5d-498e-9069-560e0c933704","Type":"ContainerStarted","Data":"e10e3cdd95473dc85cf5af544adb2b7e695d2afbf5689363d33ea2f75d5c6bcd"}
Oct 30 03:09:34.751857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:34.751635    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:34.817878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:34.817830    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" podStartSLOduration=2.421289647 podStartE2EDuration="2.817817015s" totalImagesPullingTime="396.527368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:34.816383269 +0000 UTC m=+1713.890554076" watchObservedRunningTime="2026-10-30 03:09:34.817817015 +0000 UTC m=+1713.891987804"
Oct 30 03:09:36.588485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:36.588452    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:09:37.599066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:37.599040    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:09:39.186288 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:39.186252    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:09:40.768545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:40.768519    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:09:44.495479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:44.495448    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:09:44.496001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:44.495651    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" podUID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" containerName="maas-api" containerID="cri-o://3c317f68c6fb4c9326890418ac38a56a8e1306708b329d10efb3dc7ecd5682b4" gracePeriod=30
Oct 30 03:09:45.161549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.161513    4214 generic.go:356] "Generic (PLEG): container finished" podID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" containerID="3c317f68c6fb4c9326890418ac38a56a8e1306708b329d10efb3dc7ecd5682b4" exitCode=0
Oct 30 03:09:45.190531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.190497    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:09:45.290594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.290550    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\" (UniqueName: \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\") pod \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") "
Oct 30 03:09:45.290992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.290955    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\") pod \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") "
Oct 30 03:09:45.291724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.291688    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca" pod "d8a20fad-af87-46e2-a6bc-f2c09fc60872" (UID: "d8a20fad-af87-46e2-a6bc-f2c09fc60872"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:45.292320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.292298    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\") pod \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") "
Oct 30 03:09:45.292505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.292494    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\") pod \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\" (UID: \"d8a20fad-af87-46e2-a6bc-f2c09fc60872\") "
Oct 30 03:09:45.293568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.293544    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a20fad-af87-46e2-a6bc-f2c09fc60872-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:45.317652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.315018    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg" pod "d8a20fad-af87-46e2-a6bc-f2c09fc60872" (UID: "d8a20fad-af87-46e2-a6bc-f2c09fc60872"). InnerVolumeSpecName "kube-api-access-rxdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:45.322292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.318989    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs" pod "d8a20fad-af87-46e2-a6bc-f2c09fc60872" (UID: "d8a20fad-af87-46e2-a6bc-f2c09fc60872"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:45.352257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.343373    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls" pod "d8a20fad-af87-46e2-a6bc-f2c09fc60872" (UID: "d8a20fad-af87-46e2-a6bc-f2c09fc60872"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:45.352257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.343821    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" event={"ID":"d8a20fad-af87-46e2-a6bc-f2c09fc60872","Type":"ContainerDied","Data":"3c317f68c6fb4c9326890418ac38a56a8e1306708b329d10efb3dc7ecd5682b4"}
Oct 30 03:09:45.352257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.343888    4214 scope.go:122] "RemoveContainer" containerID="3c317f68c6fb4c9326890418ac38a56a8e1306708b329d10efb3dc7ecd5682b4"
Oct 30 03:09:45.396209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.396175    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:45.396209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.396210    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a20fad-af87-46e2-a6bc-f2c09fc60872-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:45.396418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:45.396224    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdvg\" (UniqueName: \"kubernetes.io/projected/d8a20fad-af87-46e2-a6bc-f2c09fc60872-kube-api-access-rxdvg\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:46.133571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.133534    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:09:46.139276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.134292    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" containerID="cri-o://7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36" gracePeriod=30
Oct 30 03:09:46.158225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.158195    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:09:46.159200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.159169    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" containerID="cri-o://9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b" gracePeriod=30
Oct 30 03:09:46.167757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.167734    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:09:46.168048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.168020    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" containerID="cri-o://9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d" gracePeriod=30
Oct 30 03:09:46.181436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.181411    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:09:46.181628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.181608    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" containerID="cri-o://4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d" gracePeriod=30
Oct 30 03:09:46.186826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.186804    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"
Oct 30 03:09:46.186902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.186839    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp" event={"ID":"d8a20fad-af87-46e2-a6bc-f2c09fc60872","Type":"ContainerDied","Data":"e754fc65c7335c964f7a9124d5218ef8cd8921e6fbc4b21091463dc878dacf77"}
Oct 30 03:09:46.415302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.412888    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:09:46.519984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:46.512633    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3337a-698fbc577d-j9mvp"]
Oct 30 03:09:47.075793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:47.075754    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" path="/var/lib/kubelet/pods/d8a20fad-af87-46e2-a6bc-f2c09fc60872/volumes"
Oct 30 03:09:47.414606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:47.414530    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:47.414962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:47.414601    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:48.447568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:48.447534    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.448044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:48.447590    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:49.162300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:49.160980    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:49.162300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:49.161039    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:51.149466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.149417    4214 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:51.150070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.149504    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:51.593017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.592918    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:09:51.598103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.598073    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:09:51.666848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.664443    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:09:51.667563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.667537    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:09:51.774901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.774804    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.774901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.774873    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.774932    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.774961    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775046    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775120    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775155    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775239    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775277    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775308    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.775518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775366    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.775518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775518    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775560    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775604    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775635    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775677    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775727    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775764    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775801    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.775866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775845    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775921    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775951    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.775986    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776062    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776127    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\") pod \"ed907564-a568-401b-b9fc-df7dddf72873\" (UID: \"ed907564-a568-401b-b9fc-df7dddf72873\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776212    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776275    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776343    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776386    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776450    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776512    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776557    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\") pod \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\" (UID: \"2e5d19a2-81ff-4e61-8e21-cdb01f814b39\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776591    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776654    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\") pod \"ce74a684-df53-4681-94e8-01802098f1b4\" (UID: \"ce74a684-df53-4681-94e8-01802098f1b4\") "
Oct 30 03:09:51.776896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.776727    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\") pod \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\" (UID: \"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40\") "
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777422    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777669    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777823    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777957    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777913    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.777918    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.778169    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.778389    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.778891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.778618    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.783200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.780725    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.783200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.782116    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.783200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.782395    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.783200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.782581    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.783200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.782759    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.783786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.783765    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.784082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.784063    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.785535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.785513    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.788609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.788335    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.793265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.793198    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.793265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.793254    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.793565 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.793542    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.794120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.793771    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.794223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.794138    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.795219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.795037    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.795344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.795293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.795981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.795942    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.798128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.798092    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "kube-api-access-n9tb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.798662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.798589    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.798928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.798898    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "kube-api-access-89fkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.799409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.799375    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.799814 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.799786    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.800020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.799987    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8" pod "ce74a684-df53-4681-94e8-01802098f1b4" (UID: "ce74a684-df53-4681-94e8-01802098f1b4"). InnerVolumeSpecName "kube-api-access-lfcr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.801427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.801365    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.801805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.801781    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "kube-api-access-p7vr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.802349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.802326    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo" pod "ed907564-a568-401b-b9fc-df7dddf72873" (UID: "ed907564-a568-401b-b9fc-df7dddf72873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.802641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.802621    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.804908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.804886    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy" pod "2e5d19a2-81ff-4e61-8e21-cdb01f814b39" (UID: "2e5d19a2-81ff-4e61-8e21-cdb01f814b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.804999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.804978    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy" pod "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" (UID: "3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.879550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879517    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879547    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879556    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879565    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879574    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879581    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879590    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879600    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fkx\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-kube-api-access-89fkx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879609    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879618    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce74a684-df53-4681-94e8-01802098f1b4-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879626    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed907564-a568-401b-b9fc-df7dddf72873-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879634    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879641    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879648    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879656    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed907564-a568-401b-b9fc-df7dddf72873-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879664    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879672    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9tb7\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-kube-api-access-n9tb7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879680    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879688    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879697    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879704    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879712    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce74a684-df53-4681-94e8-01802098f1b4-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879720    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879727    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879737    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879751    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879765    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d19a2-81ff-4e61-8e21-cdb01f814b39-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879775    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfcr8\" (UniqueName: \"kubernetes.io/projected/ce74a684-df53-4681-94e8-01802098f1b4-kube-api-access-lfcr8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879784    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce74a684-df53-4681-94e8-01802098f1b4-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879792    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879801    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed907564-a568-401b-b9fc-df7dddf72873-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879809    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed907564-a568-401b-b9fc-df7dddf72873-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879817    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.879847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:51.879824    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vr2\" (UniqueName: \"kubernetes.io/projected/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40-kube-api-access-p7vr2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.241988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.241954    4214 generic.go:356] "Generic (PLEG): container finished" podID="ce74a684-df53-4681-94e8-01802098f1b4" containerID="7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36" exitCode=0
Oct 30 03:09:52.242411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.241998    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"
Oct 30 03:09:52.242411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.242045    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerDied","Data":"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"}
Oct 30 03:09:52.242411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.242109    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75" event={"ID":"ce74a684-df53-4681-94e8-01802098f1b4","Type":"ContainerDied","Data":"3bfe6f3c63132a584a967b6cec9e0ea46810b4701dde4548fffae949590ad2c7"}
Oct 30 03:09:52.242411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.242138    4214 scope.go:122] "RemoveContainer" containerID="7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"
Oct 30 03:09:52.244088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.244064    4214 generic.go:356] "Generic (PLEG): container finished" podID="ed907564-a568-401b-b9fc-df7dddf72873" containerID="9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d" exitCode=0
Oct 30 03:09:52.244245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.244117    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"
Oct 30 03:09:52.244245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.244150    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" event={"ID":"ed907564-a568-401b-b9fc-df7dddf72873","Type":"ContainerDied","Data":"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"}
Oct 30 03:09:52.244245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.244211    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s" event={"ID":"ed907564-a568-401b-b9fc-df7dddf72873","Type":"ContainerDied","Data":"941838f85c786957afe411942f462bbe2e5ec4ddd59098e976b7519ed9324246"}
Oct 30 03:09:52.246048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.246025    4214 generic.go:356] "Generic (PLEG): container finished" podID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerID="9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b" exitCode=0
Oct 30 03:09:52.246293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.246274    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" event={"ID":"2e5d19a2-81ff-4e61-8e21-cdb01f814b39","Type":"ContainerDied","Data":"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"}
Oct 30 03:09:52.246373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.246319    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6" event={"ID":"2e5d19a2-81ff-4e61-8e21-cdb01f814b39","Type":"ContainerDied","Data":"0a3679dcc4d126aee897938fad13e32c6396b936a7aab0775bf7f4b535ccdb44"}
Oct 30 03:09:52.246439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.246384    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"
Oct 30 03:09:52.254337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.254009    4214 generic.go:356] "Generic (PLEG): container finished" podID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerID="4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d" exitCode=0
Oct 30 03:09:52.254337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.254051    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" event={"ID":"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40","Type":"ContainerDied","Data":"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"}
Oct 30 03:09:52.254337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.254086    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"
Oct 30 03:09:52.254609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.254089    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs" event={"ID":"3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40","Type":"ContainerDied","Data":"e87db7db90d8a2ae349a41750b37442d6266a627551464fbfe98fecb2b8c5237"}
Oct 30 03:09:52.259452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.259432    4214 scope.go:122] "RemoveContainer" containerID="708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"
Oct 30 03:09:52.273534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.273513    4214 scope.go:122] "RemoveContainer" containerID="7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"
Oct 30 03:09:52.273813 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:52.273794    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36\": container with ID starting with 7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36 not found: ID does not exist" containerID="7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"
Oct 30 03:09:52.273895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.273820    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36"} err="failed to get container status \"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36\": rpc error: code = NotFound desc = could not find container \"7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36\": container with ID starting with 7baccc5651fcad791dc6bf8da69ec3b338a974a944b101da9eb6dd101caa8a36 not found: ID does not exist"
Oct 30 03:09:52.273895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.273836    4214 scope.go:122] "RemoveContainer" containerID="708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"
Oct 30 03:09:52.274090 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:52.274070    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5\": container with ID starting with 708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5 not found: ID does not exist" containerID="708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"
Oct 30 03:09:52.274152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.274096    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5"} err="failed to get container status \"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5\": rpc error: code = NotFound desc = could not find container \"708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5\": container with ID starting with 708e07043ed9d4feedb850a2e5f221a563875bad64bdc0050fd76dbc833bf7e5 not found: ID does not exist"
Oct 30 03:09:52.274152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.274119    4214 scope.go:122] "RemoveContainer" containerID="9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"
Oct 30 03:09:52.288764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.288740    4214 scope.go:122] "RemoveContainer" containerID="9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"
Oct 30 03:09:52.289091 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:52.289050    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d\": container with ID starting with 9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d not found: ID does not exist" containerID="9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"
Oct 30 03:09:52.289209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.289098    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d"} err="failed to get container status \"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d\": rpc error: code = NotFound desc = could not find container \"9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d\": container with ID starting with 9d5f7f89582e190e1a682da4bcf189aba4308758990c9f63f83d464f601a844d not found: ID does not exist"
Oct 30 03:09:52.289209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.289121    4214 scope.go:122] "RemoveContainer" containerID="9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"
Oct 30 03:09:52.305450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.305428    4214 scope.go:122] "RemoveContainer" containerID="9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"
Oct 30 03:09:52.305784 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:52.305760    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b\": container with ID starting with 9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b not found: ID does not exist" containerID="9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"
Oct 30 03:09:52.305864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.305794    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b"} err="failed to get container status \"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b\": rpc error: code = NotFound desc = could not find container \"9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b\": container with ID starting with 9d3123c8f2de692fa1bc762915359cbdead007a33f3eeaf301282eb1df12927b not found: ID does not exist"
Oct 30 03:09:52.305864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.305817    4214 scope.go:122] "RemoveContainer" containerID="4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"
Oct 30 03:09:52.319889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.319870    4214 scope.go:122] "RemoveContainer" containerID="4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"
Oct 30 03:09:52.320214 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:52.320186    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d\": container with ID starting with 4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d not found: ID does not exist" containerID="4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"
Oct 30 03:09:52.320341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.320215    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d"} err="failed to get container status \"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d\": rpc error: code = NotFound desc = could not find container \"4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d\": container with ID starting with 4f493ea2f74689201967ce3ef001964eb3e0f459e90943d3d271b4936ca14c4d not found: ID does not exist"
Oct 30 03:09:52.356590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.356563    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:09:52.364478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.364429    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-gkr75"]
Oct 30 03:09:52.516278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.516180    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:09:52.521771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.521688    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-zc9cs"]
Oct 30 03:09:52.527275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.527248    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:09:52.534134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.534109    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-wrjk6"]
Oct 30 03:09:52.557287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.557262    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:09:52.563895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:52.563872    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-62k4s"]
Oct 30 03:09:53.109477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.109442    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" path="/var/lib/kubelet/pods/2e5d19a2-81ff-4e61-8e21-cdb01f814b39/volumes"
Oct 30 03:09:53.110528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.110501    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" path="/var/lib/kubelet/pods/3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40/volumes"
Oct 30 03:09:53.111195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.111172    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce74a684-df53-4681-94e8-01802098f1b4" path="/var/lib/kubelet/pods/ce74a684-df53-4681-94e8-01802098f1b4/volumes"
Oct 30 03:09:53.112368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.112345    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed907564-a568-401b-b9fc-df7dddf72873" path="/var/lib/kubelet/pods/ed907564-a568-401b-b9fc-df7dddf72873/volumes"
Oct 30 03:09:53.112821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.112802    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:09:53.113065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:53.113042    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy" containerID="cri-o://f57b3b4f031548c0a48d2aa9dce389e9ef66130a25b4fb8f9286b0ab54ff99f0" gracePeriod=30
Oct 30 03:09:57.952829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.952789    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:09:57.954008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.953987    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:09:57.954073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954012    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:09:57.954073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954044    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy"
Oct 30 03:09:57.954073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954054    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954084    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954094    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954135    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954142    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954174    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" containerName="maas-api"
Oct 30 03:09:57.954263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954181    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" containerName="maas-api"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954411    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a20fad-af87-46e2-a6bc-f2c09fc60872" containerName="maas-api"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954430    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954443    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3117b6d5-2d5b-40e7-b6a6-53cd00d8fb40" containerName="istio-proxy"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954457    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed907564-a568-401b-b9fc-df7dddf72873" containerName="istio-proxy"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954470    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:09:57.954571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.954481    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5d19a2-81ff-4e61-8e21-cdb01f814b39" containerName="istio-proxy"
Oct 30 03:09:57.959272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.959252    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:57.977612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.977582    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:09:57.983174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:57.983152    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c1fce3-serving-cert\""
Oct 30 03:09:58.050168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.050134    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.050428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.050405    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.050569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.050548    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv872\" (UniqueName: \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.050648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.050635    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.152628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.152592    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.152780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.152712    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv872\" (UniqueName: \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.152896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.152817    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.153010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.152980    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.153131 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:58.153051    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-serving-cert: secret "maas-api-e2e-inf-a-c1fce3-serving-cert" not found
Oct 30 03:09:58.153194 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:09:58.153133    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls podName:57cd6b97-58e0-4803-9145-9673538a3546 nodeName:}" failed. No retries permitted until 2026-10-30 03:09:58.653112196 +0000 UTC m=+1737.727282983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls") pod "maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" (UID: "57cd6b97-58e0-4803-9145-9673538a3546") : secret "maas-api-e2e-inf-a-c1fce3-serving-cert" not found
Oct 30 03:09:58.153485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.153453    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.155289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.155220    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.175538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.175503    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv872\" (UniqueName: \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.487024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.486985    4214 generic.go:356] "Generic (PLEG): container finished" podID="853fa3d3-0a35-4181-9432-405ba12d0589" containerID="f57b3b4f031548c0a48d2aa9dce389e9ef66130a25b4fb8f9286b0ab54ff99f0" exitCode=0
Oct 30 03:09:58.487177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.487165    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerDied","Data":"f57b3b4f031548c0a48d2aa9dce389e9ef66130a25b4fb8f9286b0ab54ff99f0"}
Oct 30 03:09:58.487318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.487205    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7" event={"ID":"853fa3d3-0a35-4181-9432-405ba12d0589","Type":"ContainerDied","Data":"9cb1c25f0f6d998c86db5cc99ddd78565edf50208e854795f57ae7d68dd7c73d"}
Oct 30 03:09:58.487318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.487222    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb1c25f0f6d998c86db5cc99ddd78565edf50208e854795f57ae7d68dd7c73d"
Oct 30 03:09:58.487318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.487257    4214 scope.go:122] "RemoveContainer" containerID="462cd71628422504745e1ac26b3ceb22037dc5970698c42314c7d874cf994495"
Oct 30 03:09:58.512596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.512571    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:09:58.663429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663394    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663485    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663526    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663638    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663679    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663731    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663799    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.663958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663897    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:58.663958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663951    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.664134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664034    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:58.664134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664092    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\") pod \"853fa3d3-0a35-4181-9432-405ba12d0589\" (UID: \"853fa3d3-0a35-4181-9432-405ba12d0589\") "
Oct 30 03:09:58.664285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664196    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:58.664340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.663970    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:58.664572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664551    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.664702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664552    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:58.664889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664872    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.664955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664894    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.664955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664908    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.664955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664922    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.664955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.664935    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/853fa3d3-0a35-4181-9432-405ba12d0589-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.666204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.666170    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:58.666518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.666487    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:58.666518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.666506    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "kube-api-access-xhl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:58.666814 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.666794    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token" pod "853fa3d3-0a35-4181-9432-405ba12d0589" (UID: "853fa3d3-0a35-4181-9432-405ba12d0589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:58.667275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.667218    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:58.766346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.766317    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/853fa3d3-0a35-4181-9432-405ba12d0589-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.766346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.766345    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.766346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.766358    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/853fa3d3-0a35-4181-9432-405ba12d0589-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.766628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.766370    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhl99\" (UniqueName: \"kubernetes.io/projected/853fa3d3-0a35-4181-9432-405ba12d0589-kube-api-access-xhl99\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:09:58.878290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:58.878256    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:09:59.264832 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:09:59.264799    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cd6b97_58e0_4803_9145_9673538a3546.slice/crio-dd0d224705d5e98ad978f5f2a6b7f37ccd36fa15d3247bf37faa6f7704674ebe WatchSource:0}: Error finding container dd0d224705d5e98ad978f5f2a6b7f37ccd36fa15d3247bf37faa6f7704674ebe: Status 404 returned error can't find the container with id dd0d224705d5e98ad978f5f2a6b7f37ccd36fa15d3247bf37faa6f7704674ebe
Oct 30 03:09:59.266548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.266528    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:09:59.267679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.267654    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:09:59.520298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.512845    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"
Oct 30 03:09:59.520298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.515998    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" event={"ID":"57cd6b97-58e0-4803-9145-9673538a3546","Type":"ContainerStarted","Data":"dd0d224705d5e98ad978f5f2a6b7f37ccd36fa15d3247bf37faa6f7704674ebe"}
Oct 30 03:09:59.599548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.599513    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:09:59.604223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:09:59.604196    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a3337a-openshift-default-6c5ddf5bdf-mcsw7"]
Oct 30 03:10:00.537556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:00.537517    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" event={"ID":"57cd6b97-58e0-4803-9145-9673538a3546","Type":"ContainerStarted","Data":"d3e0fe6864f4409b383f39524001107d6184bc1f69e464b09d017edbf2df3dcc"}
Oct 30 03:10:00.537948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:00.537613    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:10:00.580839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:00.580791    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" podStartSLOduration=3.232436559 podStartE2EDuration="3.580778183s" totalImagesPullingTime="348.341624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:00.578367598 +0000 UTC m=+1739.652538393" watchObservedRunningTime="2026-10-30 03:10:00.580778183 +0000 UTC m=+1739.654949011"
Oct 30 03:10:01.069940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.069910    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" path="/var/lib/kubelet/pods/853fa3d3-0a35-4181-9432-405ba12d0589/volumes"
Oct 30 03:10:01.735855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.735810    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:10:01.736757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736735    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:10:01.736757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736753    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce74a684-df53-4681-94e8-01802098f1b4" containerName="istio-proxy"
Oct 30 03:10:01.736893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736772    4214 cpu_manager.go:414] "containerMap: removing container" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.736893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736777    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.736893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736805    4214 cpu_manager.go:414] "containerMap: removing container" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.736893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736814    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.737113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736951    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.737113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.736969    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="853fa3d3-0a35-4181-9432-405ba12d0589" containerName="istio-proxy"
Oct 30 03:10:01.742626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.742604    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.761441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.760296    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:10:01.761588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.761439    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.761715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.761634    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.761841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.761818    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.761985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.761858    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qzjg\" (UniqueName: \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.766028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.766000    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-734d43-serving-cert\""
Oct 30 03:10:01.863088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.863054    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.863320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.863101    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qzjg\" (UniqueName: \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.863320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.863259    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.863471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.863377    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.864025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.863995    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.865888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.865863    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.866302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.866280    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:01.883256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:01.883195    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qzjg\" (UniqueName: \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\") pod \"maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:02.078540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:02.076415    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:02.394671 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:02.394639    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaabb08d_b6e7_4fce_aff8_6572a6412da4.slice/crio-5b7fd3248adc7679d52485304efab4a44a7aab37f22f8b17fc359799c415f510 WatchSource:0}: Error finding container 5b7fd3248adc7679d52485304efab4a44a7aab37f22f8b17fc359799c415f510: Status 404 returned error can't find the container with id 5b7fd3248adc7679d52485304efab4a44a7aab37f22f8b17fc359799c415f510
Oct 30 03:10:02.394855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:02.394668    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:10:02.559391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:02.559351    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" event={"ID":"daabb08d-b6e7-4fce-aff8-6572a6412da4","Type":"ContainerStarted","Data":"5b7fd3248adc7679d52485304efab4a44a7aab37f22f8b17fc359799c415f510"}
Oct 30 03:10:03.571468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:03.571432    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" event={"ID":"daabb08d-b6e7-4fce-aff8-6572a6412da4","Type":"ContainerStarted","Data":"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"}
Oct 30 03:10:03.571847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:03.571553    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:03.616406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:03.616356    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" podStartSLOduration=2.243818416 podStartE2EDuration="2.61634147s" totalImagesPullingTime="372.523054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:03.612983753 +0000 UTC m=+1742.687154546" watchObservedRunningTime="2026-10-30 03:10:03.61634147 +0000 UTC m=+1742.690512263"
Oct 30 03:10:04.180062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:04.180025    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:10:04.208491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:04.208459    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:10:06.552963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:06.552929    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:10:09.167137 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:10:09.167097    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:10:09.181425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.181360    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" podUID="b5c36f5c-ce33-464e-940d-29bef9639b7e" containerName="payload-processing" containerID="cri-o://0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a" gracePeriod=25
Oct 30 03:10:09.209138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.209105    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" podUID="5d100648-f008-4989-90c2-4cc43ac4bb67" containerName="payload-pre-processing" containerID="cri-o://f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c" gracePeriod=25
Oct 30 03:10:09.577952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.577916    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:10:09.591565 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.591540    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:10:09.598437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.598414    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:10:09.631023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.630992    4214 generic.go:356] "Generic (PLEG): container finished" podID="b5c36f5c-ce33-464e-940d-29bef9639b7e" containerID="0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a" exitCode=0
Oct 30 03:10:09.631191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.631046    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"
Oct 30 03:10:09.631191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.631070    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" event={"ID":"b5c36f5c-ce33-464e-940d-29bef9639b7e","Type":"ContainerDied","Data":"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"}
Oct 30 03:10:09.631191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.631118    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8" event={"ID":"b5c36f5c-ce33-464e-940d-29bef9639b7e","Type":"ContainerDied","Data":"e5eac79ed90ac932a3284f98be151072d198cef5bb4f372c1dfb1af028e0e759"}
Oct 30 03:10:09.631191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.631147    4214 scope.go:122] "RemoveContainer" containerID="0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"
Oct 30 03:10:09.633353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.633329    4214 generic.go:356] "Generic (PLEG): container finished" podID="5d100648-f008-4989-90c2-4cc43ac4bb67" containerID="f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c" exitCode=0
Oct 30 03:10:09.633484 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.633464    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" event={"ID":"5d100648-f008-4989-90c2-4cc43ac4bb67","Type":"ContainerDied","Data":"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"}
Oct 30 03:10:09.633551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.633503    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl" event={"ID":"5d100648-f008-4989-90c2-4cc43ac4bb67","Type":"ContainerDied","Data":"2bea9a3874ff054c764b8801513eae5cc4b811b3f9e964524206ac5782bf219e"}
Oct 30 03:10:09.633625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.633558    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"
Oct 30 03:10:09.656439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.649040    4214 scope.go:122] "RemoveContainer" containerID="0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"
Oct 30 03:10:09.656439 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:10:09.649398    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a\": container with ID starting with 0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a not found: ID does not exist" containerID="0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"
Oct 30 03:10:09.656439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.649431    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a"} err="failed to get container status \"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a\": rpc error: code = NotFound desc = could not find container \"0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a\": container with ID starting with 0c28fffa301fad88608725451086643bd4f77e147ed91131aa84ce8bb8f9f44a not found: ID does not exist"
Oct 30 03:10:09.656439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.649453    4214 scope.go:122] "RemoveContainer" containerID="f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"
Oct 30 03:10:09.680294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.679917    4214 scope.go:122] "RemoveContainer" containerID="f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"
Oct 30 03:10:09.680446 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:10:09.680348    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c\": container with ID starting with f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c not found: ID does not exist" containerID="f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"
Oct 30 03:10:09.680446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.680381    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c"} err="failed to get container status \"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c\": rpc error: code = NotFound desc = could not find container \"f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c\": container with ID starting with f5853374a81418e8048e402c7bb2df62d420e41b63e37c13036914237c04f69c not found: ID does not exist"
Oct 30 03:10:09.726538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.726455    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\" (UniqueName: \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\") pod \"b5c36f5c-ce33-464e-940d-29bef9639b7e\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") "
Oct 30 03:10:09.726707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.726575    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\") pod \"b5c36f5c-ce33-464e-940d-29bef9639b7e\" (UID: \"b5c36f5c-ce33-464e-940d-29bef9639b7e\") "
Oct 30 03:10:09.726707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.726640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\" (UniqueName: \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\") pod \"5d100648-f008-4989-90c2-4cc43ac4bb67\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") "
Oct 30 03:10:09.726992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.726960    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\") pod \"5d100648-f008-4989-90c2-4cc43ac4bb67\" (UID: \"5d100648-f008-4989-90c2-4cc43ac4bb67\") "
Oct 30 03:10:09.730356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.730327    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume" pod "5d100648-f008-4989-90c2-4cc43ac4bb67" (UID: "5d100648-f008-4989-90c2-4cc43ac4bb67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:09.731284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.731261    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc" pod "5d100648-f008-4989-90c2-4cc43ac4bb67" (UID: "5d100648-f008-4989-90c2-4cc43ac4bb67"). InnerVolumeSpecName "kube-api-access-dwfjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:09.732011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.731982    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfjc\" (UniqueName: \"kubernetes.io/projected/5d100648-f008-4989-90c2-4cc43ac4bb67-kube-api-access-dwfjc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:09.732108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.732017    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d100648-f008-4989-90c2-4cc43ac4bb67-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:09.733557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.733529    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume" pod "b5c36f5c-ce33-464e-940d-29bef9639b7e" (UID: "b5c36f5c-ce33-464e-940d-29bef9639b7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:09.734209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.734181    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk" pod "b5c36f5c-ce33-464e-940d-29bef9639b7e" (UID: "b5c36f5c-ce33-464e-940d-29bef9639b7e"). InnerVolumeSpecName "kube-api-access-mpgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:09.833246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.833208    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgtk\" (UniqueName: \"kubernetes.io/projected/b5c36f5c-ce33-464e-940d-29bef9639b7e-kube-api-access-mpgtk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:09.833246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.833249    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c36f5c-ce33-464e-940d-29bef9639b7e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:09.984286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.984183    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:10:09.989735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:09.989702    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-71f552-fd4bfdd56-vw4k8"]
Oct 30 03:10:10.062253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:10.061086    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:10:10.067250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:10.065030    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-71f552-55789dbff-bksnl"]
Oct 30 03:10:11.079249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:11.078781    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d100648-f008-4989-90c2-4cc43ac4bb67" path="/var/lib/kubelet/pods/5d100648-f008-4989-90c2-4cc43ac4bb67/volumes"
Oct 30 03:10:11.079741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:11.079686    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c36f5c-ce33-464e-940d-29bef9639b7e" path="/var/lib/kubelet/pods/b5c36f5c-ce33-464e-940d-29bef9639b7e/volumes"
Oct 30 03:10:14.868486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:14.868393    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:10:14.868915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:14.868658    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" podUID="56eb50bc-01ac-4b54-8b99-6042ef702914" containerName="maas-api" containerID="cri-o://558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28" gracePeriod=30
Oct 30 03:10:15.670249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.669980    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:10:15.702609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.702577    4214 generic.go:356] "Generic (PLEG): container finished" podID="56eb50bc-01ac-4b54-8b99-6042ef702914" containerID="558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28" exitCode=0
Oct 30 03:10:15.702774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.702636    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"
Oct 30 03:10:15.702774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.702664    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" event={"ID":"56eb50bc-01ac-4b54-8b99-6042ef702914","Type":"ContainerDied","Data":"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"}
Oct 30 03:10:15.702774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.702747    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s" event={"ID":"56eb50bc-01ac-4b54-8b99-6042ef702914","Type":"ContainerDied","Data":"fa6bf751d6ccd21af935ad6fb34ec514dabd3f601c5c56c5c73ab738fe8a4941"}
Oct 30 03:10:15.702774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.702771    4214 scope.go:122] "RemoveContainer" containerID="558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"
Oct 30 03:10:15.716105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.716032    4214 scope.go:122] "RemoveContainer" containerID="558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"
Oct 30 03:10:15.716389 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:10:15.716361    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28\": container with ID starting with 558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28 not found: ID does not exist" containerID="558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"
Oct 30 03:10:15.716570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.716543    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28"} err="failed to get container status \"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28\": rpc error: code = NotFound desc = could not find container \"558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28\": container with ID starting with 558e8908e4f45799b9e8db8bc39db509a504143eb2f6596f84e9d43c75a22f28 not found: ID does not exist"
Oct 30 03:10:15.744931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.744900    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\") pod \"56eb50bc-01ac-4b54-8b99-6042ef702914\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") "
Oct 30 03:10:15.745083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.744972    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\" (UniqueName: \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\") pod \"56eb50bc-01ac-4b54-8b99-6042ef702914\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") "
Oct 30 03:10:15.745083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.745058    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\") pod \"56eb50bc-01ac-4b54-8b99-6042ef702914\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") "
Oct 30 03:10:15.745548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.745268    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\") pod \"56eb50bc-01ac-4b54-8b99-6042ef702914\" (UID: \"56eb50bc-01ac-4b54-8b99-6042ef702914\") "
Oct 30 03:10:15.746644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.746614    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca" pod "56eb50bc-01ac-4b54-8b99-6042ef702914" (UID: "56eb50bc-01ac-4b54-8b99-6042ef702914"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:15.749393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.749356    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb" pod "56eb50bc-01ac-4b54-8b99-6042ef702914" (UID: "56eb50bc-01ac-4b54-8b99-6042ef702914"). InnerVolumeSpecName "kube-api-access-x7jwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:15.751082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.751050    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs" pod "56eb50bc-01ac-4b54-8b99-6042ef702914" (UID: "56eb50bc-01ac-4b54-8b99-6042ef702914"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:10:15.751341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.751312    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls" pod "56eb50bc-01ac-4b54-8b99-6042ef702914" (UID: "56eb50bc-01ac-4b54-8b99-6042ef702914"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:10:15.847163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.847059    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:15.847163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.847094    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eb50bc-01ac-4b54-8b99-6042ef702914-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:15.847163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.847104    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7jwb\" (UniqueName: \"kubernetes.io/projected/56eb50bc-01ac-4b54-8b99-6042ef702914-kube-api-access-x7jwb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:15.847163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:15.847115    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56eb50bc-01ac-4b54-8b99-6042ef702914-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:16.056218 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:16.056188    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:10:16.068664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:16.068636    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ab09ae-55bf997789-zhk2s"]
Oct 30 03:10:17.073455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:17.073420    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56eb50bc-01ac-4b54-8b99-6042ef702914" path="/var/lib/kubelet/pods/56eb50bc-01ac-4b54-8b99-6042ef702914/volumes"
Oct 30 03:10:18.577799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.577761    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:10:18.578912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578879    4214 cpu_manager.go:414] "containerMap: removing container" podUID="56eb50bc-01ac-4b54-8b99-6042ef702914" containerName="maas-api"
Oct 30 03:10:18.578912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578905    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eb50bc-01ac-4b54-8b99-6042ef702914" containerName="maas-api"
Oct 30 03:10:18.579120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578927    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5d100648-f008-4989-90c2-4cc43ac4bb67" containerName="payload-pre-processing"
Oct 30 03:10:18.579120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578938    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d100648-f008-4989-90c2-4cc43ac4bb67" containerName="payload-pre-processing"
Oct 30 03:10:18.579120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578953    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b5c36f5c-ce33-464e-940d-29bef9639b7e" containerName="payload-processing"
Oct 30 03:10:18.579120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.578958    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c36f5c-ce33-464e-940d-29bef9639b7e" containerName="payload-processing"
Oct 30 03:10:18.579380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.579156    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eb50bc-01ac-4b54-8b99-6042ef702914" containerName="maas-api"
Oct 30 03:10:18.579380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.579170    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d100648-f008-4989-90c2-4cc43ac4bb67" containerName="payload-pre-processing"
Oct 30 03:10:18.579380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.579178    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c36f5c-ce33-464e-940d-29bef9639b7e" containerName="payload-processing"
Oct 30 03:10:18.584875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.584842    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.595760    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.595826    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ld8\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.595879    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.595928    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.595999    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.596147    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.596200    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.596241    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.596418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.596280    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.612888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.612854    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-537001-openshift-default-dockercfg-5v6qv\""
Oct 30 03:10:18.613603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.613575    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.769952    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770020    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770085    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770113    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770170    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ld8\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770242    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770312    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770403    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.779251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.770660    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.842292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.831422    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.842292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.833333    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.842292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.833594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.842292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.833823    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.842292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.835079    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.891256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.890859    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.900762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.899981    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ld8\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.901161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.901136    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.914490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.914447    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-kh7mf\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:18.914969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:18.914948    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:19.411043 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.411014    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:10:19.411656 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:19.411625    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42204c5f_32ff_4325_89a8_ce0d537827aa.slice/crio-9f2df498b4383ca9b7807f9de88863185092085342812a871ff43dbaf9791784 WatchSource:0}: Error finding container 9f2df498b4383ca9b7807f9de88863185092085342812a871ff43dbaf9791784: Status 404 returned error can't find the container with id 9f2df498b4383ca9b7807f9de88863185092085342812a871ff43dbaf9791784
Oct 30 03:10:19.414660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.414629    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:19.414722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.414693    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:19.414756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.414724    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:19.896823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.896787    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerStarted","Data":"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"}
Oct 30 03:10:19.897328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.896850    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerStarted","Data":"9f2df498b4383ca9b7807f9de88863185092085342812a871ff43dbaf9791784"}
Oct 30 03:10:19.915282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.915205    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:19.940418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:19.940110    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podStartSLOduration=1.940093254 podStartE2EDuration="1.940093254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:19.939835721 +0000 UTC m=+1759.014006515" watchObservedRunningTime="2026-10-30 03:10:19.940093254 +0000 UTC m=+1759.014264050"
Oct 30 03:10:20.921846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:20.921817    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:21.915320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:21.915284    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:21.916314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:21.916296    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:10:24.304371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:24.304322    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:10:24.306023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:24.305768    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy" containerID="cri-o://43713bd079a64fc04b772de647a78085be5532e2cba331cc8834cbefd230569a" gracePeriod=30
Oct 30 03:10:25.424763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.424730    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:10:25.429342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.429320    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.463330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.459544    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7f767-openshift-default-dockercfg-s4d5t\""
Oct 30 03:10:25.473952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.472623    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:10:25.719303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.694279    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.719769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.719725    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.720112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.720091    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.720303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.720285    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.720451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.720435    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.720667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.720651    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.723678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.720798    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.723905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.723881    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.735161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.724104    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7gf\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.907355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.907323    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7gf\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.907661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.907639    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.907808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.907794    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908023    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908168    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908306    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908504    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908641    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.908784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.908771    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.909432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.909410    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.915466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.912053    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.915689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.912637    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.915843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.914070    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.915962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.914334    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.973509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.955782    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:25.982293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.982255    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7gf\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:26.000254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:25.999830    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:26.085057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.085001    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\") pod \"e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") " pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:26.166635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.166596    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:26.720052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.719098    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:10:26.728422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.727482    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:26.796264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.788346    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": context deadline exceeded"
Oct 30 03:10:26.820260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.819026    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:10:26.825432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.822739    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:10:26.825432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.823035    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:10:26.861275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.850019    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-537001\""
Oct 30 03:10:26.889259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.889201    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:26.889701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.889672    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgjm\" (UniqueName: \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:26.917841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:26.917802    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-537001-dockercfg-pwr5s\""
Oct 30 03:10:27.019886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.018732    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:27.019886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.019009    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgjm\" (UniqueName: \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:27.020121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.020010    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:27.112375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.095935    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgjm\" (UniqueName: \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\") pod \"payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:27.215424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.215361    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:27.533345 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:27.530665    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e8fffa_bf6d_404c_8da4_87f0e3d0a464.slice/crio-a98cf744b50860a074f493f2d30897ace4936ee1ee7abada73ddbd8eb78efa69 WatchSource:0}: Error finding container a98cf744b50860a074f493f2d30897ace4936ee1ee7abada73ddbd8eb78efa69: Status 404 returned error can't find the container with id a98cf744b50860a074f493f2d30897ace4936ee1ee7abada73ddbd8eb78efa69
Oct 30 03:10:27.573268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.540651    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:10:27.829676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.815131    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:27.829676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.815207    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:27.829676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.815253    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:27.906287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:27.906156    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerStarted","Data":"a98cf744b50860a074f493f2d30897ace4936ee1ee7abada73ddbd8eb78efa69"}
Oct 30 03:10:28.018722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.018695    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:10:28.916285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.916241    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerStarted","Data":"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"}
Oct 30 03:10:28.918282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.918252    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" event={"ID":"41e6a7af-8b52-4d61-88b3-a930ea49daf0","Type":"ContainerStarted","Data":"735a5f7c6e1ff1367e176d4e4285c6a740d34cbe073b2aee9d24ae2f5fdc6e44"}
Oct 30 03:10:28.918421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.918346    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" event={"ID":"41e6a7af-8b52-4d61-88b3-a930ea49daf0","Type":"ContainerStarted","Data":"c3fc06a5f6642b7632770da505775f2deb808bd4bdb16f4a8a668b791b67a64e"}
Oct 30 03:10:28.918502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.918476    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:28.996249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:28.995854    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podStartSLOduration=3.995834181 podStartE2EDuration="3.995834181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:28.984806369 +0000 UTC m=+1768.058977164" watchObservedRunningTime="2026-10-30 03:10:28.995834181 +0000 UTC m=+1768.070004976"
Oct 30 03:10:29.048253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.036001    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" podStartSLOduration=4.035982575 podStartE2EDuration="4.035982575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:29.033307514 +0000 UTC m=+1768.107478308" watchObservedRunningTime="2026-10-30 03:10:29.035982575 +0000 UTC m=+1768.110153370"
Oct 30 03:10:29.195337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.187908    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:29.255257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.253543    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:29.929716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.929681    4214 generic.go:356] "Generic (PLEG): container finished" podID="2f636458-995c-42ef-a64e-a2fd496443fa" containerID="43713bd079a64fc04b772de647a78085be5532e2cba331cc8834cbefd230569a" exitCode=0
Oct 30 03:10:29.930164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.929881    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerDied","Data":"43713bd079a64fc04b772de647a78085be5532e2cba331cc8834cbefd230569a"}
Oct 30 03:10:29.930164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.929955    4214 scope.go:122] "RemoveContainer" containerID="a566b47bd0b7e0d3c4f1853293de38d76de54c412280401fbaa29be100fbb4ee"
Oct 30 03:10:29.930317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.930303    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:29.931546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:29.931530    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:10:30.421364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.421331    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:10:30.530165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530131    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.530470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530447    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.530739 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530719    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.531627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530964    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.531834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.531814    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.531977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.531962    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.532154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.532138    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.532344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.532326    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.532811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530476    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:10:30.532912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.530783    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:10:30.532912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.531008    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:10:30.532912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.532790    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\") pod \"2f636458-995c-42ef-a64e-a2fd496443fa\" (UID: \"2f636458-995c-42ef-a64e-a2fd496443fa\") "
Oct 30 03:10:30.533498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.533475    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:10:30.533672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.533647    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.533757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.533680    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f636458-995c-42ef-a64e-a2fd496443fa-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.533757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.533694    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.534095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.534075    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:10:30.610580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.610436    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:30.611481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.611071    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:10:30.611481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.611272    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "kube-api-access-z58f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:10:30.612050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.612019    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy" pod "2f636458-995c-42ef-a64e-a2fd496443fa" (UID: "2f636458-995c-42ef-a64e-a2fd496443fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635201    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635276    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58f6\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-kube-api-access-z58f6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635294    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635308    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f636458-995c-42ef-a64e-a2fd496443fa-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635322    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f636458-995c-42ef-a64e-a2fd496443fa-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.668293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.635334    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f636458-995c-42ef-a64e-a2fd496443fa-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:10:30.977324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.971539    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"
Oct 30 03:10:30.979718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.979676    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf" event={"ID":"2f636458-995c-42ef-a64e-a2fd496443fa","Type":"ContainerDied","Data":"34786626bf45c04f06c365bd87de5bbdb70d4858fc5de5bcd190665b5cebba89"}
Oct 30 03:10:30.979962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:30.979945    4214 scope.go:122] "RemoveContainer" containerID="43713bd079a64fc04b772de647a78085be5532e2cba331cc8834cbefd230569a"
Oct 30 03:10:31.094273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:31.094037    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:10:31.094273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:31.094086    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ab09ae-openshift-default-5dcb9bb99f-pq7vf"]
Oct 30 03:10:33.073973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.073934    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" path="/var/lib/kubelet/pods/2f636458-995c-42ef-a64e-a2fd496443fa/volumes"
Oct 30 03:10:33.481524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.481447    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:10:33.482605 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.482584    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.482731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.482721    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.482850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.482842    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.482910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.482902    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.483157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.483145    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.483261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.483250    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f636458-995c-42ef-a64e-a2fd496443fa" containerName="istio-proxy"
Oct 30 03:10:33.489382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.489360    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.512721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.512687    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7f767\""
Oct 30 03:10:33.512871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.512824    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e7f767-dockercfg-77xhw\""
Oct 30 03:10:33.523851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.523807    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:10:33.580380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.580308    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.581078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.581012    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqvz\" (UniqueName: \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.706201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.703672    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.706201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.703960    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqvz\" (UniqueName: \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.706201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.705637    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.794318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.782437    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:10:33.872462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.872430    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqvz\" (UniqueName: \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\") pod \"payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:33.886746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.886681    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:33.905839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.905804    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:10:33.978856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.978120    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6d2g\" (UniqueName: \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:33.978856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:33.978471    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.080448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.080350    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.080955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.080503    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6d2g\" (UniqueName: \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.081831 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.081799    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.100854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.100817    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6d2g\" (UniqueName: \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\") pod \"payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.111158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.111131    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:34.212726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.212447    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:34.495290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.495211    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:10:34.498535 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:34.498502    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d15b39_6856_440f_b72e_61ab258ab2a7.slice/crio-99885e465582745e9377e311acdb3e43d7992d021cdfed76ae83f6ee35512fe9 WatchSource:0}: Error finding container 99885e465582745e9377e311acdb3e43d7992d021cdfed76ae83f6ee35512fe9: Status 404 returned error can't find the container with id 99885e465582745e9377e311acdb3e43d7992d021cdfed76ae83f6ee35512fe9
Oct 30 03:10:34.624742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:34.624712    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:10:34.626639 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:34.626594    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be1c6c8_b965_46f7_b651_755cc14dcff4.slice/crio-eb9e69484504372af7bf6b4e74b3fe7c30b23078f04259c17c3e26e87f184c74 WatchSource:0}: Error finding container eb9e69484504372af7bf6b4e74b3fe7c30b23078f04259c17c3e26e87f184c74: Status 404 returned error can't find the container with id eb9e69484504372af7bf6b4e74b3fe7c30b23078f04259c17c3e26e87f184c74
Oct 30 03:10:35.076494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.076449    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:10:35.076852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.076811    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:10:35.084986 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.084956    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" event={"ID":"76d15b39-6856-440f-b72e-61ab258ab2a7","Type":"ContainerStarted","Data":"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"}
Oct 30 03:10:35.085449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.085010    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" event={"ID":"76d15b39-6856-440f-b72e-61ab258ab2a7","Type":"ContainerStarted","Data":"99885e465582745e9377e311acdb3e43d7992d021cdfed76ae83f6ee35512fe9"}
Oct 30 03:10:35.085449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.085054    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:35.087129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.087107    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" event={"ID":"0be1c6c8-b965-46f7-b651-755cc14dcff4","Type":"ContainerStarted","Data":"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"}
Oct 30 03:10:35.087254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.087150    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" event={"ID":"0be1c6c8-b965-46f7-b651-755cc14dcff4","Type":"ContainerStarted","Data":"eb9e69484504372af7bf6b4e74b3fe7c30b23078f04259c17c3e26e87f184c74"}
Oct 30 03:10:35.087254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.087170    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:35.128819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.128775    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" podStartSLOduration=2.128761834 podStartE2EDuration="2.128761834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:35.123592374 +0000 UTC m=+1774.197763167" watchObservedRunningTime="2026-10-30 03:10:35.128761834 +0000 UTC m=+1774.202932626"
Oct 30 03:10:35.170623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:35.170565    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" podStartSLOduration=2.170546029 podStartE2EDuration="2.170546029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:35.167008291 +0000 UTC m=+1774.241179084" watchObservedRunningTime="2026-10-30 03:10:35.170546029 +0000 UTC m=+1774.244716824"
Oct 30 03:10:36.262418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:36.262372    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:10:36.262938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:36.262434    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:10:37.106395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.106365    4214 generic.go:356] "Generic (PLEG): container finished" podID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerID="c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e" exitCode=0
Oct 30 03:10:37.106654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.106444    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerDied","Data":"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"}
Oct 30 03:10:37.107017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.106983    4214 scope.go:122] "RemoveContainer" containerID="c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.109128    4214 generic.go:356] "Generic (PLEG): container finished" podID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerID="700456c4b704c8e1157a78fe402c309806a9feefd8f01e7e72e0f0e24dfea7ee" exitCode=0
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.109462    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerDied","Data":"700456c4b704c8e1157a78fe402c309806a9feefd8f01e7e72e0f0e24dfea7ee"}
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.109560    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.109641    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.109688    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:37.110432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.110031    4214 scope.go:122] "RemoveContainer" containerID="700456c4b704c8e1157a78fe402c309806a9feefd8f01e7e72e0f0e24dfea7ee"
Oct 30 03:10:37.114475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.114443    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:37.114580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.114524    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:37.114580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:37.114562    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:38.120516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:38.120477    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerStarted","Data":"13f400683f5471704a6ec21b5e18958b190e84ff02874d07a8540db3a0958d9e"}
Oct 30 03:10:38.122645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:38.122617    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerStarted","Data":"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"}
Oct 30 03:10:38.473117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:38.472625    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:10:38.473117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:38.472751    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:38.473117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:38.472795    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:39.021415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.021100    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:10:39.021415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.021131    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:39.021415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.021172    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:39.410540 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:10:39.408587    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:10:39.485254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.485109    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:39.485254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.485170    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:39.951500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:39.951460    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:10:40.160255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:40.134504    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:40.160255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:40.134559    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:40.473251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:40.472621    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:40.473251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:40.472680    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:41.021283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:41.021248    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:41.021545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:41.021521    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:41.473251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:41.472869    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:41.473251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:41.472942    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:42.023185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:42.023060    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:42.023185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:42.023116    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:42.487839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:42.487801    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:42.488471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:42.488440    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:43.021550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:43.021508    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:43.021726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:43.021569    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:43.488318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:43.482024    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:43.488318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:43.482083    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:44.023683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:44.023647    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 03:10:44.024158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:44.024077    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 03:10:44.473810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:44.473767    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 03:10:44.474193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:44.474141    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 03:10:45.026469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.026439    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:10:45.026962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.026935    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:10:45.027612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.027588    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:10:45.478839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.478805    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:10:45.479687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.479660    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:10:45.480527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:45.480510    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:10:46.098533 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:46.098504    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:10:46.098951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:46.098578    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:10:52.710586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.710546    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:10:52.719671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.719638    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:52.734366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.734340    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:10:52.738324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.738302    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4f415d-dockercfg-bknkp\""
Oct 30 03:10:52.738595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.738571    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4f415d\""
Oct 30 03:10:52.900254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.899606    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:52.900254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:52.899937    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phs8t\" (UniqueName: \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.062397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.056650    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:10:53.079569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.077980    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phs8t\" (UniqueName: \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.079569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.078130    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.079569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.079080    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.097189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.097146    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:10:53.100720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.099891    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.101838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.101819    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phs8t\" (UniqueName: \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\") pod \"payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.280515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.280474    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8zk\" (UniqueName: \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.280915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.280887    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.340208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.340120    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:53.382410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.382375    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8zk\" (UniqueName: \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.382925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.382613    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.383572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.383548    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.404603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.404543    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8zk\" (UniqueName: \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\") pod \"payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.459343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.454671    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:53.686086 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:53.686041    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd31f5ac_ee3d_4d70_b477_5c37d72bbb67.slice/crio-c9e23182605962fe2390f85cb7a339e98b00f4593d324a0f26313eba65d9b609 WatchSource:0}: Error finding container c9e23182605962fe2390f85cb7a339e98b00f4593d324a0f26313eba65d9b609: Status 404 returned error can't find the container with id c9e23182605962fe2390f85cb7a339e98b00f4593d324a0f26313eba65d9b609
Oct 30 03:10:53.694344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.689595    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:10:53.810462 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:10:53.810435    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c237a67_034f_4fb7_8423_790f9269bf55.slice/crio-52e936dfa61e7cc306f636ff7563ab268c450770f598c88a467e7a7c4341eb6c WatchSource:0}: Error finding container 52e936dfa61e7cc306f636ff7563ab268c450770f598c88a467e7a7c4341eb6c: Status 404 returned error can't find the container with id 52e936dfa61e7cc306f636ff7563ab268c450770f598c88a467e7a7c4341eb6c
Oct 30 03:10:53.810791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:53.810519    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:10:54.674703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.674149    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" event={"ID":"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67","Type":"ContainerStarted","Data":"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"}
Oct 30 03:10:54.674703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.674218    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" event={"ID":"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67","Type":"ContainerStarted","Data":"c9e23182605962fe2390f85cb7a339e98b00f4593d324a0f26313eba65d9b609"}
Oct 30 03:10:54.675993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.675481    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:10:54.679717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.679630    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" event={"ID":"6c237a67-034f-4fb7-8423-790f9269bf55","Type":"ContainerStarted","Data":"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"}
Oct 30 03:10:54.679954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.679938    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:10:54.680071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.680056    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" event={"ID":"6c237a67-034f-4fb7-8423-790f9269bf55","Type":"ContainerStarted","Data":"52e936dfa61e7cc306f636ff7563ab268c450770f598c88a467e7a7c4341eb6c"}
Oct 30 03:10:54.718027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.717967    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" podStartSLOduration=2.717949017 podStartE2EDuration="2.717949017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:54.716637271 +0000 UTC m=+1793.790808065" watchObservedRunningTime="2026-10-30 03:10:54.717949017 +0000 UTC m=+1793.792119811"
Oct 30 03:10:54.757495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:54.757451    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" podStartSLOduration=2.757436688 podStartE2EDuration="2.757436688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:54.755581213 +0000 UTC m=+1793.829752009" watchObservedRunningTime="2026-10-30 03:10:54.757436688 +0000 UTC m=+1793.831607516"
Oct 30 03:10:55.690065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.690032    4214 generic.go:356] "Generic (PLEG): container finished" podID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerID="480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91" exitCode=0
Oct 30 03:10:55.690734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.690121    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerDied","Data":"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"}
Oct 30 03:10:55.690828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.690765    4214 scope.go:122] "RemoveContainer" containerID="480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"
Oct 30 03:10:55.692628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.692603    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:55.692706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.692677    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:55.692743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:55.692710    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:10:56.701330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:56.701293    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerStarted","Data":"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"}
Oct 30 03:10:57.211611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:57.211512    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:10:57.211850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:57.211821    4214 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 03:10:57.211997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:57.211874    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 03:10:58.212022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:58.211985    4214 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 03:10:58.212453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:58.212036    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 03:10:59.211389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:59.211350    4214 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 03:10:59.211598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:10:59.211409    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 03:11:00.211725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:00.211689    4214 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 03:11:00.212139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:00.211743    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 03:11:00.455345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:00.455284    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:11:00.471675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:00.471585    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:11:01.212073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:01.211627    4214 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 03:11:01.212073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:01.211701    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 03:11:02.175946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:02.175910    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:11:02.217013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:02.216983    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:11:02.790203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:02.790172    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:11:05.456035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.455979    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" podUID="adfb82d5-1f38-41dd-9882-6068e161846f" containerName="payload-processing" containerID="cri-o://3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d" gracePeriod=25
Oct 30 03:11:05.472935 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.472892    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" podUID="7e8aab19-1f5c-425f-a76f-d2e861657c88" containerName="payload-pre-processing" containerID="cri-o://5d3cb410f708b5cfaa9cdee4377a566a0e2b4e56d72cc72e0b215a7bbdba5abc" gracePeriod=25
Oct 30 03:11:05.694369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.694341    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:11:05.823682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.823651    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.829343    4214 generic.go:356] "Generic (PLEG): container finished" podID="7e8aab19-1f5c-425f-a76f-d2e861657c88" containerID="5d3cb410f708b5cfaa9cdee4377a566a0e2b4e56d72cc72e0b215a7bbdba5abc" exitCode=0
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.829526    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" event={"ID":"7e8aab19-1f5c-425f-a76f-d2e861657c88","Type":"ContainerDied","Data":"5d3cb410f708b5cfaa9cdee4377a566a0e2b4e56d72cc72e0b215a7bbdba5abc"}
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.831545    4214 generic.go:356] "Generic (PLEG): container finished" podID="adfb82d5-1f38-41dd-9882-6068e161846f" containerID="3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d" exitCode=0
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.831727    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" event={"ID":"adfb82d5-1f38-41dd-9882-6068e161846f","Type":"ContainerDied","Data":"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"}
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.831773    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d" event={"ID":"adfb82d5-1f38-41dd-9882-6068e161846f","Type":"ContainerDied","Data":"4d9845ead71e0db6281d061ce8eb6a82271e96b4a1b37a11654f9ef3f917ede2"}
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.831811    4214 scope.go:122] "RemoveContainer" containerID="3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"
Oct 30 03:11:05.832047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.832002    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"
Oct 30 03:11:05.893480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.891063    4214 scope.go:122] "RemoveContainer" containerID="3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"
Oct 30 03:11:05.896223 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:05.896186    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d\": container with ID starting with 3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d not found: ID does not exist" containerID="3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"
Oct 30 03:11:05.896223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.896262    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d"} err="failed to get container status \"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d\": rpc error: code = NotFound desc = could not find container \"3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d\": container with ID starting with 3db82e5fd8ffc6cf1f64b91d9ba6a6e1f8b8b0a470ce6867119dc4c284739d9d not found: ID does not exist"
Oct 30 03:11:05.900303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.900281    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:11:05.939168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.939142    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\" (UniqueName: \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\") pod \"adfb82d5-1f38-41dd-9882-6068e161846f\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") "
Oct 30 03:11:05.939392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.939375    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\") pod \"adfb82d5-1f38-41dd-9882-6068e161846f\" (UID: \"adfb82d5-1f38-41dd-9882-6068e161846f\") "
Oct 30 03:11:05.939836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.939813    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume" pod "adfb82d5-1f38-41dd-9882-6068e161846f" (UID: "adfb82d5-1f38-41dd-9882-6068e161846f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:05.940130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.940106    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfb82d5-1f38-41dd-9882-6068e161846f-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:05.941905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:05.941883    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr" pod "adfb82d5-1f38-41dd-9882-6068e161846f" (UID: "adfb82d5-1f38-41dd-9882-6068e161846f"). InnerVolumeSpecName "kube-api-access-x87vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:06.000461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.000382    4214 scope.go:122] "RemoveContainer" containerID="2e477c2e28c76a333ebb2b05756177f3bdced4cb75b2c7bfcd3bb01293183db5"
Oct 30 03:11:06.040922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.040895    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\") pod \"7e8aab19-1f5c-425f-a76f-d2e861657c88\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") "
Oct 30 03:11:06.041022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.040980    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\" (UniqueName: \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\") pod \"7e8aab19-1f5c-425f-a76f-d2e861657c88\" (UID: \"7e8aab19-1f5c-425f-a76f-d2e861657c88\") "
Oct 30 03:11:06.041299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.041278    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume" pod "7e8aab19-1f5c-425f-a76f-d2e861657c88" (UID: "7e8aab19-1f5c-425f-a76f-d2e861657c88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:06.041909 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.041885    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87vr\" (UniqueName: \"kubernetes.io/projected/adfb82d5-1f38-41dd-9882-6068e161846f-kube-api-access-x87vr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:06.042016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.041913    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e8aab19-1f5c-425f-a76f-d2e861657c88-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:06.043493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.043467    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l" pod "7e8aab19-1f5c-425f-a76f-d2e861657c88" (UID: "7e8aab19-1f5c-425f-a76f-d2e861657c88"). InnerVolumeSpecName "kube-api-access-kq95l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:06.143632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.143597    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq95l\" (UniqueName: \"kubernetes.io/projected/7e8aab19-1f5c-425f-a76f-d2e861657c88-kube-api-access-kq95l\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:06.181903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.181872    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:11:06.188417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.188395    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-67440d-766b4cb948-jtg5d"]
Oct 30 03:11:06.233675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.233648    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:11:06.234638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234605    4214 cpu_manager.go:414] "containerMap: removing container" podUID="adfb82d5-1f38-41dd-9882-6068e161846f" containerName="payload-processing"
Oct 30 03:11:06.234638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234628    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfb82d5-1f38-41dd-9882-6068e161846f" containerName="payload-processing"
Oct 30 03:11:06.234861 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234689    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7e8aab19-1f5c-425f-a76f-d2e861657c88" containerName="payload-pre-processing"
Oct 30 03:11:06.234861 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234699    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8aab19-1f5c-425f-a76f-d2e861657c88" containerName="payload-pre-processing"
Oct 30 03:11:06.235027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234959    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8aab19-1f5c-425f-a76f-d2e861657c88" containerName="payload-pre-processing"
Oct 30 03:11:06.235027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.234991    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfb82d5-1f38-41dd-9882-6068e161846f" containerName="payload-processing"
Oct 30 03:11:06.248005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.247985    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.266661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.264352    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-537001-serving-cert\""
Oct 30 03:11:06.284248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.283633    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:11:06.347243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.347204    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.347414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.347277    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.347414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.347394    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.347550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.347533    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gf2\" (UniqueName: \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.448824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.448789    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.449017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.448866    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.449017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.448908    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.449017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.449007    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gf2\" (UniqueName: \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.449975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.449945    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.452459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.452436    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.452459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.452451    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.506063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.499682    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gf2\" (UniqueName: \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\") pod \"maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.575251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.573206    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:06.705824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.705007    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:11:06.844069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.843978    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j" event={"ID":"7e8aab19-1f5c-425f-a76f-d2e861657c88","Type":"ContainerDied","Data":"05f53c480da49995a32ca91adb4160bc740be23a960e934beba23839f404c397"}
Oct 30 03:11:06.844069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.844028    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"
Oct 30 03:11:06.844069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.844042    4214 scope.go:122] "RemoveContainer" containerID="5d3cb410f708b5cfaa9cdee4377a566a0e2b4e56d72cc72e0b215a7bbdba5abc"
Oct 30 03:11:06.914677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.914645    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:11:06.920435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.920416    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-67440d-547489cbdc-j8d4j"]
Oct 30 03:11:06.924043 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:11:06.924016    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f445be_95b2_4059_8d5f_a06d13f5a84f.slice/crio-a032491eb369cc6a551189c5a63f42a6aff380ff5c19091a2e12dc51a762c566 WatchSource:0}: Error finding container a032491eb369cc6a551189c5a63f42a6aff380ff5c19091a2e12dc51a762c566: Status 404 returned error can't find the container with id a032491eb369cc6a551189c5a63f42a6aff380ff5c19091a2e12dc51a762c566
Oct 30 03:11:06.928803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:06.928781    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:11:07.071821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.071792    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8aab19-1f5c-425f-a76f-d2e861657c88" path="/var/lib/kubelet/pods/7e8aab19-1f5c-425f-a76f-d2e861657c88/volumes"
Oct 30 03:11:07.072203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.072188    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfb82d5-1f38-41dd-9882-6068e161846f" path="/var/lib/kubelet/pods/adfb82d5-1f38-41dd-9882-6068e161846f/volumes"
Oct 30 03:11:07.856414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.856375    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" event={"ID":"e0f445be-95b2-4059-8d5f-a06d13f5a84f","Type":"ContainerStarted","Data":"7b12f47307eb642111b267db44316d305af68a55aff9c26fde4cf19ec87840a6"}
Oct 30 03:11:07.856928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.856438    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" event={"ID":"e0f445be-95b2-4059-8d5f-a06d13f5a84f","Type":"ContainerStarted","Data":"a032491eb369cc6a551189c5a63f42a6aff380ff5c19091a2e12dc51a762c566"}
Oct 30 03:11:07.856928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.856471    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:07.899378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:07.899317    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" podStartSLOduration=1.392112105 podStartE2EDuration="1.899298955s" totalImagesPullingTime="507.18685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:07.898549373 +0000 UTC m=+1806.972720166" watchObservedRunningTime="2026-10-30 03:11:07.899298955 +0000 UTC m=+1806.973469748"
Oct 30 03:11:09.168368 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:09.168330    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:11:10.287692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:10.287653    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:11:13.872161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:13.872134    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:11:16.216160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.216122    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:11:16.216668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.216346    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" podUID="9082263c-60c8-4cb4-9932-a1f626544281" containerName="maas-api" containerID="cri-o://7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606" gracePeriod=30
Oct 30 03:11:16.884804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.884426    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:11:16.940885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.937978    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\" (UniqueName: \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\") pod \"9082263c-60c8-4cb4-9932-a1f626544281\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") "
Oct 30 03:11:16.941312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.941291    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\") pod \"9082263c-60c8-4cb4-9932-a1f626544281\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") "
Oct 30 03:11:16.941730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.941716    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\") pod \"9082263c-60c8-4cb4-9932-a1f626544281\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") "
Oct 30 03:11:16.942169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.942157    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\") pod \"9082263c-60c8-4cb4-9932-a1f626544281\" (UID: \"9082263c-60c8-4cb4-9932-a1f626544281\") "
Oct 30 03:11:16.942793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.940816    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl" pod "9082263c-60c8-4cb4-9932-a1f626544281" (UID: "9082263c-60c8-4cb4-9932-a1f626544281"). InnerVolumeSpecName "kube-api-access-lptwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:16.942890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.942050    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca" pod "9082263c-60c8-4cb4-9932-a1f626544281" (UID: "9082263c-60c8-4cb4-9932-a1f626544281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:16.943357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.943338    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9082263c-60c8-4cb4-9932-a1f626544281-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:16.943455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.943446    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lptwl\" (UniqueName: \"kubernetes.io/projected/9082263c-60c8-4cb4-9932-a1f626544281-kube-api-access-lptwl\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:16.944007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.943990    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs" pod "9082263c-60c8-4cb4-9932-a1f626544281" (UID: "9082263c-60c8-4cb4-9932-a1f626544281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:16.944569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:16.944549    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls" pod "9082263c-60c8-4cb4-9932-a1f626544281" (UID: "9082263c-60c8-4cb4-9932-a1f626544281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:17.022727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.022692    4214 generic.go:356] "Generic (PLEG): container finished" podID="9082263c-60c8-4cb4-9932-a1f626544281" containerID="7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606" exitCode=0
Oct 30 03:11:17.022892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.022762    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"
Oct 30 03:11:17.022892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.022771    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" event={"ID":"9082263c-60c8-4cb4-9932-a1f626544281","Type":"ContainerDied","Data":"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"}
Oct 30 03:11:17.022892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.022842    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc" event={"ID":"9082263c-60c8-4cb4-9932-a1f626544281","Type":"ContainerDied","Data":"73e82ae5eb9218e114a89b9127538c961237ce94d8445e9f1121182a4efed224"}
Oct 30 03:11:17.022892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.022868    4214 scope.go:122] "RemoveContainer" containerID="7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"
Oct 30 03:11:17.034845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.034828    4214 scope.go:122] "RemoveContainer" containerID="7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"
Oct 30 03:11:17.035117 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:17.035087    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606\": container with ID starting with 7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606 not found: ID does not exist" containerID="7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"
Oct 30 03:11:17.035266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.035124    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606"} err="failed to get container status \"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606\": rpc error: code = NotFound desc = could not find container \"7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606\": container with ID starting with 7159a0036672a7f53985ecdea7bc3a400a33ca65d54b2f7066429baa16c5d606 not found: ID does not exist"
Oct 30 03:11:17.051439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.051209    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:17.051439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.051330    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9082263c-60c8-4cb4-9932-a1f626544281-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:17.097017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.096987    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:11:17.104793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:17.104765    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-71f552-7c5975bd4f-r86rc"]
Oct 30 03:11:19.076411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:19.076380    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9082263c-60c8-4cb4-9932-a1f626544281" path="/var/lib/kubelet/pods/9082263c-60c8-4cb4-9932-a1f626544281/volumes"
Oct 30 03:11:25.314718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.314674    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" containerID="cri-o://946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63" gracePeriod=45
Oct 30 03:11:25.728280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.728207    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:11:25.765998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.765965    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:11:25.766263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.766200    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy" containerID="cri-o://9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc" gracePeriod=30
Oct 30 03:11:25.837285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.837192    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.837646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.837472    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:25.837809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.837788    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.837948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.837932    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.838147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.838127    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.838358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.838339    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.838596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.838576    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\" (UniqueName: \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\") pod \"0b09d75b-1ca8-4cf1-9526-069f23e20138\" (UID: \"0b09d75b-1ca8-4cf1-9526-069f23e20138\") "
Oct 30 03:11:25.839791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.837997    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:25.839791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.839710    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:25.840558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.840530    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:25.840658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.840563    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:25.840658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.840577    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:25.842905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.842560    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:25.850040 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.850004    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "kube-api-access-fdzjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:25.862210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.862181    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm" pod "0b09d75b-1ca8-4cf1-9526-069f23e20138" (UID: "0b09d75b-1ca8-4cf1-9526-069f23e20138"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:25.950270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.942493    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b09d75b-1ca8-4cf1-9526-069f23e20138-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:25.950270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.942527    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09d75b-1ca8-4cf1-9526-069f23e20138-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:25.950270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:25.942541    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzjl\" (UniqueName: \"kubernetes.io/projected/0b09d75b-1ca8-4cf1-9526-069f23e20138-kube-api-access-fdzjl\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:26.140732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.140652    4214 generic.go:356] "Generic (PLEG): container finished" podID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerID="946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63" exitCode=0
Oct 30 03:11:26.140967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.140928    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" event={"ID":"0b09d75b-1ca8-4cf1-9526-069f23e20138","Type":"ContainerDied","Data":"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"}
Oct 30 03:11:26.141010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.140969    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" event={"ID":"0b09d75b-1ca8-4cf1-9526-069f23e20138","Type":"ContainerDied","Data":"129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc"}
Oct 30 03:11:26.141010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.140992    4214 scope.go:122] "RemoveContainer" containerID="946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"
Oct 30 03:11:26.141186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.141168    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"
Oct 30 03:11:26.157963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.157884    4214 scope.go:122] "RemoveContainer" containerID="946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"
Oct 30 03:11:26.158342 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:26.158309    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63\": container with ID starting with 946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63 not found: ID does not exist" containerID="946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"
Oct 30 03:11:26.158451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.158349    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63"} err="failed to get container status \"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63\": rpc error: code = NotFound desc = could not find container \"946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63\": container with ID starting with 946c63dd6528cda89499fc8d5776223ad9e872d8766d4c1e6877fe390a330f63 not found: ID does not exist"
Oct 30 03:11:26.178350 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:26.178322    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b09d75b_1ca8_4cf1_9526_069f23e20138.slice/crio-129b2cb0de20b921579193b532f0b0ae3c17361b5addc626bcb355c057934ecc\": RecentStats: unable to find data in memory cache]"
Oct 30 03:11:26.233458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.233428    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:11:26.241682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.241658    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn"]
Oct 30 03:11:26.717198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:26.717157    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d30e9f/test-model-d30e9f-kserve-58c4b94c89-mfwzn" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:11:27.071793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.071759    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" path="/var/lib/kubelet/pods/0b09d75b-1ca8-4cf1-9526-069f23e20138/volumes"
Oct 30 03:11:27.196015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.195981    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:11:27.196983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.196954    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9082263c-60c8-4cb4-9932-a1f626544281" containerName="maas-api"
Oct 30 03:11:27.196983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.196974    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9082263c-60c8-4cb4-9932-a1f626544281" containerName="maas-api"
Oct 30 03:11:27.197193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.197025    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main"
Oct 30 03:11:27.197193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.197035    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main"
Oct 30 03:11:27.197375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.197262    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b09d75b-1ca8-4cf1-9526-069f23e20138" containerName="main"
Oct 30 03:11:27.197375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.197287    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9082263c-60c8-4cb4-9932-a1f626544281" containerName="maas-api"
Oct 30 03:11:27.203205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.203184    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.252355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.250401    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f21058-openshift-default-dockercfg-wb4zh\""
Oct 30 03:11:27.255449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.255414    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:11:27.366301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.361358    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.382361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.382319    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4l54\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.382693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.382667    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.382857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.382841    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.382984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.382967    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.383315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.383292    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.383505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.383484    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.383656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.383637    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.383937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.383903    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.521984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.521513    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.522269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.522245    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.523113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.523093    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.528066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.528043    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.528635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.528605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4l54\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.528778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.522685    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.528836    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.529102    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.529333    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.537501    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.537537    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.537862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.537744    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.538180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.538154    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.538440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.538422    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.539010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.538981    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.562884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.550200    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.562884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.553838    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4l54\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.562884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.554778    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\") pod \"e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") " pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:27.820951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:27.820919    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:28.158941 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:11:28.158909    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5ac6e4_1bad_4b53_9a91_fee6b1942195.slice/crio-9fe7c95f7a568fa281ef6ca2bc307404b228944b453ba0c1276dac5de1baed03 WatchSource:0}: Error finding container 9fe7c95f7a568fa281ef6ca2bc307404b228944b453ba0c1276dac5de1baed03: Status 404 returned error can't find the container with id 9fe7c95f7a568fa281ef6ca2bc307404b228944b453ba0c1276dac5de1baed03
Oct 30 03:11:28.159660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.159634    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:11:28.162217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.162184    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:28.162351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.162265    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:28.162351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.162296    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:28.300691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.298318    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": read tcp 10.133.0.2:47664->10.133.0.136:15021: read: connection reset by peer" start-of-body=
Oct 30 03:11:28.300691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:28.298428    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": read tcp 10.133.0.2:47664->10.133.0.136:15021: read: connection reset by peer"
Oct 30 03:11:29.178577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.178537    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerStarted","Data":"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"}
Oct 30 03:11:29.179016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.178603    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerStarted","Data":"9fe7c95f7a568fa281ef6ca2bc307404b228944b453ba0c1276dac5de1baed03"}
Oct 30 03:11:29.180269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.180245    4214 generic.go:356] "Generic (PLEG): container finished" podID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerID="017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658" exitCode=0
Oct 30 03:11:29.180269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.180257    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerDied","Data":"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"}
Oct 30 03:11:29.180638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.180625    4214 scope.go:122] "RemoveContainer" containerID="017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"
Oct 30 03:11:29.182387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.182360    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:29.182465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.182428    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:29.182509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.182469    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:29.242887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.242826    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podStartSLOduration=2.24280896 podStartE2EDuration="2.24280896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:29.242788347 +0000 UTC m=+1828.316959145" watchObservedRunningTime="2026-10-30 03:11:29.24280896 +0000 UTC m=+1828.316979754"
Oct 30 03:11:29.830521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.824570    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:29.862257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:29.861610    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:30.191520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.191440    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerStarted","Data":"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"}
Oct 30 03:11:30.191937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.191884    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:30.192781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.192757    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:11:30.463611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.463514    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:11:30.464384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.464258    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:30.468581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:30.464578    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:31.183089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.183059    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:11:31.190805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.190778    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.190946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.190877    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.190946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.190905    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.190946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.190933    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.190954    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191005    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191044    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191073    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191141    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\") pod \"31de71a5-1ac8-450e-9cde-10f8a66098a5\" (UID: \"31de71a5-1ac8-450e-9cde-10f8a66098a5\") "
Oct 30 03:11:31.191431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191149    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:31.191431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191193    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:31.191914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191433    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:31.191914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191553    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:31.191914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.191664    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:31.192096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.192039    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31de71a5-1ac8-450e-9cde-10f8a66098a5-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.192096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.192060    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.192096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.192075    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.192325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.192106    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.192325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.192127    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.193963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.193932    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "kube-api-access-z448d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:31.194267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.194220    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:11:31.194355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.194285    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:31.194494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.194474    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token" pod "31de71a5-1ac8-450e-9cde-10f8a66098a5" (UID: "31de71a5-1ac8-450e-9cde-10f8a66098a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:31.204153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.204124    4214 generic.go:356] "Generic (PLEG): container finished" podID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerID="9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc" exitCode=0
Oct 30 03:11:31.205125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.205103    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"
Oct 30 03:11:31.206353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.206322    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerDied","Data":"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"}
Oct 30 03:11:31.206464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.206383    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q" event={"ID":"31de71a5-1ac8-450e-9cde-10f8a66098a5","Type":"ContainerDied","Data":"0afce980f5bcbce181e506a11aa0d8a4afd7a6303d07c1281c308aa7c40816a6"}
Oct 30 03:11:31.206464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.206415    4214 scope.go:122] "RemoveContainer" containerID="9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"
Oct 30 03:11:31.234475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.234355    4214 scope.go:122] "RemoveContainer" containerID="7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"
Oct 30 03:11:31.287823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.287644    4214 scope.go:122] "RemoveContainer" containerID="9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"
Oct 30 03:11:31.290057 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:31.290010    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc\": container with ID starting with 9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc not found: ID does not exist" containerID="9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"
Oct 30 03:11:31.290193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.290064    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc"} err="failed to get container status \"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc\": rpc error: code = NotFound desc = could not find container \"9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc\": container with ID starting with 9d2db1ff05996fda35af7a8d5c0de228674072feea91e551efad1ea31bbdc5cc not found: ID does not exist"
Oct 30 03:11:31.290193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.290092    4214 scope.go:122] "RemoveContainer" containerID="7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"
Oct 30 03:11:31.290525 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:31.290496    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0\": container with ID starting with 7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0 not found: ID does not exist" containerID="7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"
Oct 30 03:11:31.290610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.290531    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0"} err="failed to get container status \"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0\": rpc error: code = NotFound desc = could not find container \"7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0\": container with ID starting with 7b776043a445f6a223bd43be2f9a11039a9c6b62eba5861b4bb9502350fb60a0 not found: ID does not exist"
Oct 30 03:11:31.295154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.295127    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.295154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.295153    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.295311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.295164    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z448d\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-kube-api-access-z448d\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.295311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.295174    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31de71a5-1ac8-450e-9cde-10f8a66098a5-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:31.304564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.304543    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:11:31.311223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.309407    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-71f552-openshift-default-665776fb8c-8j94q"]
Oct 30 03:11:31.465333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.458542    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:31.465333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:31.458608    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:32.462254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:32.459582    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:32.462254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:32.459649    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:33.074150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.074120    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" path="/var/lib/kubelet/pods/31de71a5-1ac8-450e-9cde-10f8a66098a5/volumes"
Oct 30 03:11:33.214066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.214025    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:11:33.214872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.214852    4214 cpu_manager.go:414] "containerMap: removing container" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:33.214872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.214869    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:33.215083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.214896    4214 cpu_manager.go:414] "containerMap: removing container" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:33.215083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.214901    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:33.215083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.215052    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:33.219008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.218984    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.276166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.274829    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:11:33.309782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.309754    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-360818-openshift-default-dockercfg-clvks\""
Oct 30 03:11:33.324445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324370    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324407    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324512    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324569    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324596    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszrq\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324657    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324688    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324718    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.324999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.324739    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426052    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426099    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426215    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426434    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426475    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426503    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426518    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426603    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426697    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426747    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszrq\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426821    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.426920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.426850    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.427046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.427029    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.427099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.427039    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.428984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.428961    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.429548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.429522    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.467265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.461487    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:33.467265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.461536    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:33.475858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.475823    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.476019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.475922    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszrq\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.555250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.548532    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:33.914825 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:11:33.914779    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf2ede2_ce90_4134_b94d_80aa6a60a5a2.slice/crio-6c995fec3e0bd59b3e8addddf8e8a585204e44919ad114c8b8ce5dc4b105f05a WatchSource:0}: Error finding container 6c995fec3e0bd59b3e8addddf8e8a585204e44919ad114c8b8ce5dc4b105f05a: Status 404 returned error can't find the container with id 6c995fec3e0bd59b3e8addddf8e8a585204e44919ad114c8b8ce5dc4b105f05a
Oct 30 03:11:33.915576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.915542    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:11:33.919307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.919276    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:33.919381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.919345    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:33.919381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:33.919372    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:11:34.396317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.392137    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:11:34.398289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.398220    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="31de71a5-1ac8-450e-9cde-10f8a66098a5" containerName="istio-proxy"
Oct 30 03:11:34.406633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.403083    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.518100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.518060    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97r4\" (UniqueName: \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.518658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.518128    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.518658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.518264    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.518658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.518408    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.526524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.519464    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:34.526524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.519511    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:34.527491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.527087    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4f415d-serving-cert\""
Oct 30 03:11:34.600380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.600307    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:11:34.602541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.602375    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" event={"ID":"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2","Type":"ContainerStarted","Data":"6c995fec3e0bd59b3e8addddf8e8a585204e44919ad114c8b8ce5dc4b105f05a"}
Oct 30 03:11:34.620147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.620119    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97r4\" (UniqueName: \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.620300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.620168    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.620300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.620255    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.620457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.620302    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.620969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.620944    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.621122 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:34.621099    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-serving-cert: secret "maas-api-e2e-inf-b-4f415d-serving-cert" not found
Oct 30 03:11:34.621196 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:34.621166    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls podName:ed7e6cb1-a252-4add-aee1-b44344b8095d nodeName:}" failed. No retries permitted until 2026-10-30 03:11:35.121146465 +0000 UTC m=+1834.195317253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls") pod "maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" (UID: "ed7e6cb1-a252-4add-aee1-b44344b8095d") : secret "maas-api-e2e-inf-b-4f415d-serving-cert" not found
Oct 30 03:11:34.624103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.624072    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:34.679305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:34.679269    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97r4\" (UniqueName: \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:35.149251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.140800    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:35.207254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.205807    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:35.463317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.461841    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 03:11:35.463317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.461907    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 03:11:35.502074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.502045    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:35.565252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.563480    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:35.583814 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.566392    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:35.583814 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.566442    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:35.616632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.615617    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" event={"ID":"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2","Type":"ContainerStarted","Data":"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"}
Oct 30 03:11:35.855707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.855653    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podStartSLOduration=2.855637747 podStartE2EDuration="2.855637747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:34.675492626 +0000 UTC m=+1833.749663420" watchObservedRunningTime="2026-10-30 03:11:35.855637747 +0000 UTC m=+1834.929808545"
Oct 30 03:11:35.858375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:35.858347    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:11:35.859943 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:11:35.859914    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7e6cb1_a252_4add_aee1_b44344b8095d.slice/crio-b958c90fe4c22dc3c3e61ea347ed5381b4445435d1cfbee40ae64e3da896b77f WatchSource:0}: Error finding container b958c90fe4c22dc3c3e61ea347ed5381b4445435d1cfbee40ae64e3da896b77f: Status 404 returned error can't find the container with id b958c90fe4c22dc3c3e61ea347ed5381b4445435d1cfbee40ae64e3da896b77f
Oct 30 03:11:36.559258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.555505    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:36.559258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.555571    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:36.619965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.618701    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:11:36.620457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.620268    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:11:36.621141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.621118    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:11:36.631540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.631506    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" event={"ID":"ed7e6cb1-a252-4add-aee1-b44344b8095d","Type":"ContainerStarted","Data":"71906b304185b720b5a0564f79eee59efba72b94fd853a4a317120d494c81c6f"}
Oct 30 03:11:36.631677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.631552    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" event={"ID":"ed7e6cb1-a252-4add-aee1-b44344b8095d","Type":"ContainerStarted","Data":"b958c90fe4c22dc3c3e61ea347ed5381b4445435d1cfbee40ae64e3da896b77f"}
Oct 30 03:11:36.631677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.631591    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:36.725403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:36.725352    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" podStartSLOduration=2.241929302 podStartE2EDuration="2.725337952s" totalImagesPullingTime="483.40865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:36.723421478 +0000 UTC m=+1835.797592271" watchObservedRunningTime="2026-10-30 03:11:36.725337952 +0000 UTC m=+1835.799508745"
Oct 30 03:11:37.555253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:37.554626    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:37.555253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:37.554687    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:38.555570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:38.555534    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:38.556145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:38.556115    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:39.196112 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:39.196074    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:11:39.566287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:39.559531    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:39.566287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:39.559592    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:40.556622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:40.556584    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:40.556905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:40.556877    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:40.796271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:40.794631    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:11:41.552414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:41.552373    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:11:41.552605 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:41.552445    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:11:42.557859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:42.557819    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:42.558378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:42.558327    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:42.559141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:42.559119    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:11:42.683639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:42.683608    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:11:51.572882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:51.572798    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:11:56.133169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.133133    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" podUID="be83b073-8355-4439-b9bf-a38114aa7484" containerName="main" containerID="cri-o://e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c" gracePeriod=45
Oct 30 03:11:56.587998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.587923    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing" containerID="cri-o://6f6ced2f503e5f631a25439eef554c6e6304807947b5913241a10e479f3793a4" gracePeriod=25
Oct 30 03:11:56.830489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.830464    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.896893    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\" (UniqueName: \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.897036    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.897167    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\" (UniqueName: \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.897295    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.897335    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.897398    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\") pod \"be83b073-8355-4439-b9bf-a38114aa7484\" (UID: \"be83b073-8355-4439-b9bf-a38114aa7484\") "
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.898532    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:56.900466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.898728    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:56.909287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.903548    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:56.926734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.924515    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:56.926734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.924776    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "kube-api-access-psfdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:56.997418    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs" pod "be83b073-8355-4439-b9bf-a38114aa7484" (UID: "be83b073-8355-4439-b9bf-a38114aa7484"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015134    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015171    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015187    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015201    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psfdl\" (UniqueName: \"kubernetes.io/projected/be83b073-8355-4439-b9bf-a38114aa7484-kube-api-access-psfdl\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015213    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be83b073-8355-4439-b9bf-a38114aa7484-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.015278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.015242    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be83b073-8355-4439-b9bf-a38114aa7484-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:11:57.073257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.072417    4214 generic.go:356] "Generic (PLEG): container finished" podID="be83b073-8355-4439-b9bf-a38114aa7484" containerID="e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c" exitCode=0
Oct 30 03:11:57.073257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.072599    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"
Oct 30 03:11:57.122169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.122125    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" event={"ID":"be83b073-8355-4439-b9bf-a38114aa7484","Type":"ContainerDied","Data":"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"}
Oct 30 03:11:57.122169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.122170    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g" event={"ID":"be83b073-8355-4439-b9bf-a38114aa7484","Type":"ContainerDied","Data":"d0d2a1f54f105742364bedc718db74a7d47da9383c89b4b234174374e7c297eb"}
Oct 30 03:11:57.122442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.122192    4214 scope.go:122] "RemoveContainer" containerID="e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"
Oct 30 03:11:57.138069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.137969    4214 scope.go:122] "RemoveContainer" containerID="e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"
Oct 30 03:11:57.138681 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:11:57.138658    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c\": container with ID starting with e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c not found: ID does not exist" containerID="e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"
Oct 30 03:11:57.138791 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.138707    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c"} err="failed to get container status \"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c\": rpc error: code = NotFound desc = could not find container \"e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c\": container with ID starting with e602654bdf605dcfac1737fb12900946e0fdc79164bb113cdfce25925f4c6e2c not found: ID does not exist"
Oct 30 03:11:57.167818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.167734    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:11:57.172945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:57.172909    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cea203/test-model-cea203-kserve-845dc5785f-2747g"]
Oct 30 03:11:59.011252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:59.010624    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:11:59.081762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:11:59.081727    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be83b073-8355-4439-b9bf-a38114aa7484" path="/var/lib/kubelet/pods/be83b073-8355-4439-b9bf-a38114aa7484/volumes"
Oct 30 03:12:00.974398 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:00.974341    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:12:00.975070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:00.975031    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:04.300792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.300759    4214 generic.go:356] "Generic (PLEG): container finished" podID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerID="43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d" exitCode=0
Oct 30 03:12:04.301383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.301007    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerDied","Data":"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"}
Oct 30 03:12:04.301542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.301520    4214 scope.go:122] "RemoveContainer" containerID="43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"
Oct 30 03:12:04.303640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.303608    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:04.303749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.303681    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:04.303749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:04.303716    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:05.341300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:05.341256    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerStarted","Data":"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"}
Oct 30 03:12:06.072427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.072385    4214 scope.go:122] "RemoveContainer" containerID="c77bdf3a96e8f8cd1828eb3fbaef98ef30b6c95ff455a6d74ef18d31205e523d"
Oct 30 03:12:06.183710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.183667    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:12:06.183924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.183905    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:06.183993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.183957    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:06.789018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.788978    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:12:06.804653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:06.804614    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" podUID="0ab12aee-96be-4959-990d-8233cd65da07" containerName="maas-api" containerID="cri-o://c82f1e1bdf3d45f473d877ab4ce7c8c5c711c45de434ec45e21ffd931571c493" gracePeriod=30
Oct 30 03:12:07.213296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:07.210736    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:07.213296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:07.210824    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:07.479302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:07.474749    4214 generic.go:356] "Generic (PLEG): container finished" podID="0ab12aee-96be-4959-990d-8233cd65da07" containerID="c82f1e1bdf3d45f473d877ab4ce7c8c5c711c45de434ec45e21ffd931571c493" exitCode=0
Oct 30 03:12:07.479302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:07.475089    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" event={"ID":"0ab12aee-96be-4959-990d-8233cd65da07","Type":"ContainerDied","Data":"c82f1e1bdf3d45f473d877ab4ce7c8c5c711c45de434ec45e21ffd931571c493"}
Oct 30 03:12:07.882299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:07.881038    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:12:08.009874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.002117    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\") pod \"0ab12aee-96be-4959-990d-8233cd65da07\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") "
Oct 30 03:12:08.009874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.002377    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\" (UniqueName: \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\") pod \"0ab12aee-96be-4959-990d-8233cd65da07\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") "
Oct 30 03:12:08.009874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.002459    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\") pod \"0ab12aee-96be-4959-990d-8233cd65da07\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") "
Oct 30 03:12:08.009874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.002650    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\") pod \"0ab12aee-96be-4959-990d-8233cd65da07\" (UID: \"0ab12aee-96be-4959-990d-8233cd65da07\") "
Oct 30 03:12:08.023792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.023433    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:12:08.023792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.023508    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:08.044294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.031117    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs" pod "0ab12aee-96be-4959-990d-8233cd65da07" (UID: "0ab12aee-96be-4959-990d-8233cd65da07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:08.044294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.031915    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca" pod "0ab12aee-96be-4959-990d-8233cd65da07" (UID: "0ab12aee-96be-4959-990d-8233cd65da07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:08.044561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.044528    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn" pod "0ab12aee-96be-4959-990d-8233cd65da07" (UID: "0ab12aee-96be-4959-990d-8233cd65da07"). InnerVolumeSpecName "kube-api-access-5rgsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:08.105208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.103506    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls" pod "0ab12aee-96be-4959-990d-8233cd65da07" (UID: "0ab12aee-96be-4959-990d-8233cd65da07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:08.105208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.105120    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgsn\" (UniqueName: \"kubernetes.io/projected/0ab12aee-96be-4959-990d-8233cd65da07-kube-api-access-5rgsn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:08.105208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.105150    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:08.105208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.105164    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab12aee-96be-4959-990d-8233cd65da07-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:08.105208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.105179    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab12aee-96be-4959-990d-8233cd65da07-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:08.196255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.195994    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:08.196255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.196055    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:08.489566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.489537    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"
Oct 30 03:12:08.499268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.489531    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v" event={"ID":"0ab12aee-96be-4959-990d-8233cd65da07","Type":"ContainerDied","Data":"dbdae0e64589196cdd4beead6c9356726be06ea5928ba3d4021eb7608f97f0eb"}
Oct 30 03:12:08.499268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.489683    4214 scope.go:122] "RemoveContainer" containerID="c82f1e1bdf3d45f473d877ab4ce7c8c5c711c45de434ec45e21ffd931571c493"
Oct 30 03:12:08.580101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.580066    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:12:08.592552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.592520    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-67440d-f8867d6f5-mhs9v"]
Oct 30 03:12:08.988552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:08.988115    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:12:09.078573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.078540    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab12aee-96be-4959-990d-8233cd65da07" path="/var/lib/kubelet/pods/0ab12aee-96be-4959-990d-8233cd65da07/volumes"
Oct 30 03:12:09.183687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.183645    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:09.183932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.183717    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:09.195115 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:09.195067    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:12:09.500937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.500906    4214 generic.go:356] "Generic (PLEG): container finished" podID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerID="f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2" exitCode=0
Oct 30 03:12:09.501150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.500985    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerDied","Data":"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"}
Oct 30 03:12:09.501523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.501501    4214 scope.go:122] "RemoveContainer" containerID="f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"
Oct 30 03:12:09.504042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.503996    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:09.504331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.504293    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:09.504475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:09.504452    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:10.076474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.072601    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:12:10.149250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.148858    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0ab12aee-96be-4959-990d-8233cd65da07" containerName="maas-api"
Oct 30 03:12:10.149250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.148888    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab12aee-96be-4959-990d-8233cd65da07" containerName="maas-api"
Oct 30 03:12:10.149250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.148932    4214 cpu_manager.go:414] "containerMap: removing container" podUID="be83b073-8355-4439-b9bf-a38114aa7484" containerName="main"
Oct 30 03:12:10.149250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.148940    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be83b073-8355-4439-b9bf-a38114aa7484" containerName="main"
Oct 30 03:12:10.149250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.149204    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab12aee-96be-4959-990d-8233cd65da07" containerName="maas-api"
Oct 30 03:12:10.187250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.186278    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="be83b073-8355-4439-b9bf-a38114aa7484" containerName="main"
Oct 30 03:12:10.206772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.203296    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:10.206772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.203378    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:10.300253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.274973    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:12:10.300253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.276167    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.333305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.327760    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-100ffd-openshift-default-dockercfg-db8dw\""
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382372    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382564    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382700    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382768    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382813    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpwq\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.382860    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.383031    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.383075    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.388263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.383160    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.484343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.484312    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.484643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.484627    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.484745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.484736    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.484862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.484852    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485078    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485102    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485289    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485466    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485499    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485692    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485790    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485850    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.485923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.485891    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpwq\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.486204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.486175    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.487646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.487621    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.488297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.488273    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.506639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.506605    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.509415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.509391    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpwq\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.515945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.515903    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerStarted","Data":"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"}
Oct 30 03:12:10.759832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.759458    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:10.919250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:10.918390    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:12:11.024601 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.024515    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:11.044333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.043538    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:11.212889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.212188    4214 patch_prober.go:29] interesting pod/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:12:11.212889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.212258    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:12:11.231637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.231602    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:12:11.312313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.293449    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:12:11.312313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.293506    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:12:11.314253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.312673    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.321950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.321926    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:12:11.322079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.322053    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.484257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.433105    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643423    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643499    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643589    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxfr\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643673    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643850    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.643988    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644055    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644089    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644274    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644352    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644414    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644479    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644523    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644566    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644683    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cxm\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644745    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.646677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.644791    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.760674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.760623    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.760931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.760909    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.761094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.761077    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.761269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.761243    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxfr\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.761461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.761442    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.761748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.761723    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.762029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762009    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.762193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762165    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.762358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.762578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762554    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.762744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762727    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.762905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.762878    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.763058 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.763041    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.763222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.763206    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.764151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.764129    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.764411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.764390    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cxm\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.764602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.764580    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.764776 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.764752    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.765953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.765929    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.766764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.766744    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.767221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.767199    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.768002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.767980    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.778496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.774003    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.778496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.774713    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.779444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.779419    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.779843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.779816    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.781555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.781151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.804982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.804922    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.820426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.817999    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.820426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.819672    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.820426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.820258    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.828473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.821823    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxfr\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.828473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.822382    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.828473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.825001    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cxm\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.828473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.826766    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.913084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.911640    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:11.913084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.911797    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:11.913084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.911882    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:11.916418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.916304    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:11.916418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.916358    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:11.918860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.918823    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:11.924712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.924677    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:12:11.977688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.968679    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:11.982939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:11.982906    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:12.425512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.425429    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:12:12.481315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.481254    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:12:12.501560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.501189    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.508376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.502442    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.508376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.502507    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.508376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.502340    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.508376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.503104    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.508376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.503159    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:12.580783 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.580759    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:12:12.583979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.583951    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:12:12.586591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.586568    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:12:12.925646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.925608    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:12.925845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.925671    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:12.942482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.939913    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerStarted","Data":"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"}
Oct 30 03:12:12.942482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:12.939987    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerStarted","Data":"bfdac6d3c54ea531a0ffa6e535811481da88bf23bf45bb54250118e9f8cd02c7"}
Oct 30 03:12:13.022890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.022838    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podStartSLOduration=4.022820365 podStartE2EDuration="4.022820365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:13.020067636 +0000 UTC m=+1872.094238429" watchObservedRunningTime="2026-10-30 03:12:13.022820365 +0000 UTC m=+1872.096991161"
Oct 30 03:12:13.025358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.025279    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" event={"ID":"b74f76ad-f1e6-406e-add6-14a7f48429bb","Type":"ContainerStarted","Data":"064d04bb37b6df1cf2348cbf4946c851df25085c23c03429ac5e161f8147cf6f"}
Oct 30 03:12:13.030311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.029968    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" event={"ID":"abbd3dcf-9b5b-431d-84b2-aa207d044e74","Type":"ContainerStarted","Data":"af63ffa7d0e1ae6e29d8a423b85901d1dcb95b0cac9cdd0e20accc247a360caf"}
Oct 30 03:12:13.765252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.765176    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:13.777396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.777366    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:13.919471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.919436    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:13.919746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:13.919716    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:14.093253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.093105    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" event={"ID":"b74f76ad-f1e6-406e-add6-14a7f48429bb","Type":"ContainerStarted","Data":"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"}
Oct 30 03:12:14.184254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.171547    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" event={"ID":"abbd3dcf-9b5b-431d-84b2-aa207d044e74","Type":"ContainerStarted","Data":"a6f4689a226495600663cb9dbbb26bb9ab3afc8a40f6518c21af80ac6118f7b5"}
Oct 30 03:12:14.184254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.172971    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:14.184254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.175211    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:12:14.272254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.271632    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podStartSLOduration=4.27160992 podStartE2EDuration="4.27160992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:14.173610109 +0000 UTC m=+1873.247780903" watchObservedRunningTime="2026-10-30 03:12:14.27160992 +0000 UTC m=+1873.345780715"
Oct 30 03:12:14.283273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.282606    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podStartSLOduration=4.282585106 podStartE2EDuration="4.282585106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:14.221647614 +0000 UTC m=+1873.295818410" watchObservedRunningTime="2026-10-30 03:12:14.282585106 +0000 UTC m=+1873.356755900"
Oct 30 03:12:14.949254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.940704    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:14.949254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.940769    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.989826    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.989898    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.990032    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.990079    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.990161    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.004255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:14.990183    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:15.427251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.426332    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:15.453254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.450615    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 30 03:12:15.922253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.921948    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.922253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.922022    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:15.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.985730    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.985796    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:15.986252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.986206    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.986888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:15.986268    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:16.726451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.726388    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:16.863763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.782436    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:12:16.880842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.819119    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:16.922188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.922149    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.922411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.922219    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:12:16.998322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.984402    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.998322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.984464    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:16.998322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.984402    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.998322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:16.984527    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:17.876259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:17.863267    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:12:18.577867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.005623    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:18.578465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.578429    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:18.578588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.427307    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:18.578688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.454818    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:12:18.588671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.005663    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:18.588912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.588883    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:18.589036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.039293    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:18.590391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.590370    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:12:18.782205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.782128    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:12:18.802428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.789942    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:18.803198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.803168    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\" (UniqueName: \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\") pod \"86020f30-a736-4491-88b8-2ca91aaa570f\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") "
Oct 30 03:12:18.807250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.803550    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\") pod \"86020f30-a736-4491-88b8-2ca91aaa570f\" (UID: \"86020f30-a736-4491-88b8-2ca91aaa570f\") "
Oct 30 03:12:18.936811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.936760    4214 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:12:18.936993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.936823    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:12:18.956247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.802795    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:12:19.006927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.959408    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:12:19.032457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.032385    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:12:19.102194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:18.982637    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume" pod "86020f30-a736-4491-88b8-2ca91aaa570f" (UID: "86020f30-a736-4491-88b8-2ca91aaa570f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:19.111665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.111634    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs" pod "86020f30-a736-4491-88b8-2ca91aaa570f" (UID: "86020f30-a736-4491-88b8-2ca91aaa570f"). InnerVolumeSpecName "kube-api-access-t8bqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:19.162029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.161987    4214 generic.go:356] "Generic (PLEG): container finished" podID="86020f30-a736-4491-88b8-2ca91aaa570f" containerID="6f6ced2f503e5f631a25439eef554c6e6304807947b5913241a10e479f3793a4" exitCode=0
Oct 30 03:12:19.162327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.162287    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy" containerID="cri-o://5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7" gracePeriod=30
Oct 30 03:12:19.170489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.170458    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:19.170695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.170664    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:19.171668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.171642    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:19.171816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.171796    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:19.182628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.182599    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"
Oct 30 03:12:19.209014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.206619    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86020f30-a736-4491-88b8-2ca91aaa570f-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:19.209014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.206652    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bqs\" (UniqueName: \"kubernetes.io/projected/86020f30-a736-4491-88b8-2ca91aaa570f-kube-api-access-t8bqs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:19.245253 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:19.243984    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86020f30_a736_4491_88b8_2ca91aaa570f.slice/crio-a838c0ba44b8c7b06bcabed3e07fa64528f730369190d1b61ce022cece5d7291\": RecentStats: unable to find data in memory cache]"
Oct 30 03:12:19.290289 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:19.278124    4214 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86020f30_a736_4491_88b8_2ca91aaa570f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86020f30_a736_4491_88b8_2ca91aaa570f.slice/crio-a838c0ba44b8c7b06bcabed3e07fa64528f730369190d1b61ce022cece5d7291\": RecentStats: unable to find data in memory cache]"
Oct 30 03:12:19.311597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.311562    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:12:19.319866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.319838    4214 cpu_manager.go:414] "containerMap: removing container" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing"
Oct 30 03:12:19.320054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.320040    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing"
Oct 30 03:12:19.320638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.320615    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" containerName="payload-processing"
Oct 30 03:12:19.428206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.428110    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:12:19.428511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.428490    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.428721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.428605    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:12:19.428895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.428864    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82" event={"ID":"86020f30-a736-4491-88b8-2ca91aaa570f","Type":"ContainerDied","Data":"6f6ced2f503e5f631a25439eef554c6e6304807947b5913241a10e479f3793a4"}
Oct 30 03:12:19.429016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.429006    4214 scope.go:122] "RemoveContainer" containerID="6f6ced2f503e5f631a25439eef554c6e6304807947b5913241a10e479f3793a4"
Oct 30 03:12:19.429130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.429108    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:12:19.462283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.461808    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-100ffd\""
Oct 30 03:12:19.466252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.464912    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-100ffd-dockercfg-bcz7r\""
Oct 30 03:12:19.522557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.522527    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:12:19.530115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.529670    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.530684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.530653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ksc\" (UniqueName: \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.533083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.533053    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d30e9f-5f9c9d8546-b4z82"]
Oct 30 03:12:19.632784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.632746    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ksc\" (UniqueName: \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.633303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.633047    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.634139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.634113    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.801637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.801360    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ksc\" (UniqueName: \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\") pod \"payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") " pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:19.816580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:19.813620    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:20.015561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.015521    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:20.015736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.015584    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:20.016211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.016178    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:20.016349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.016241    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:20.507013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.506985    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:12:20.984101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.984062    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 03:12:20.984548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.984114    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 03:12:20.984548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.984065    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 03:12:20.984548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:20.984208    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 03:12:21.072217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.072173    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86020f30-a736-4491-88b8-2ca91aaa570f" path="/var/lib/kubelet/pods/86020f30-a736-4491-88b8-2ca91aaa570f/volumes"
Oct 30 03:12:21.231276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.231214    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" event={"ID":"3f58dc2c-02c8-42a1-86ae-d4c513b30263","Type":"ContainerStarted","Data":"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"}
Oct 30 03:12:21.231460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.231317    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:21.231460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.231342    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" event={"ID":"3f58dc2c-02c8-42a1-86ae-d4c513b30263","Type":"ContainerStarted","Data":"963096ff1e09437ff9bdd3b64dea9209d591d1d4dbd1094b9ae0329c0ecc99bf"}
Oct 30 03:12:21.285718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.285580    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" podStartSLOduration=3.285557277 podStartE2EDuration="3.285557277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:21.281776405 +0000 UTC m=+1880.355947270" watchObservedRunningTime="2026-10-30 03:12:21.285557277 +0000 UTC m=+1880.359728072"
Oct 30 03:12:21.988894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.988865    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:21.989397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.989377    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:21.990055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.990032    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:12:21.990162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.990095    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:21.990374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.990351    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:21.991106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:21.991088    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:12:24.540399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.540370    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:12:24.703877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.703794    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.703877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.703854    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.703887    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.703917    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704001    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704062    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704154    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704267    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704451    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\") pod \"844932dc-68d5-495b-9be3-b9023e0ba1be\" (UID: \"844932dc-68d5-495b-9be3-b9023e0ba1be\") "
Oct 30 03:12:24.704554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704260    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:24.704554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704400    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:24.704554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704523    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:24.704710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704607    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:24.704836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.704807    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:24.705398 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.705372    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844932dc-68d5-495b-9be3-b9023e0ba1be-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.705524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.705400    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.705524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.705415    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.705524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.705430    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.705524 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.705443    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.706501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.706468    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:24.706627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.706514    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:24.706627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.706548    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:24.707021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.707004    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv" pod "844932dc-68d5-495b-9be3-b9023e0ba1be" (UID: "844932dc-68d5-495b-9be3-b9023e0ba1be"). InnerVolumeSpecName "kube-api-access-kj5gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:24.807607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.807573    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.807607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.807602    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.807607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.807611    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.807845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:24.807620    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5gv\" (UniqueName: \"kubernetes.io/projected/844932dc-68d5-495b-9be3-b9023e0ba1be-kube-api-access-kj5gv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:25.241001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.240963    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:12:25.241879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.241856    4214 cpu_manager.go:414] "containerMap: removing container" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.241879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.241874    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.242068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.241911    4214 cpu_manager.go:414] "containerMap: removing container" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.242068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.241916    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.242068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.242057    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.242068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.242069    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerName="istio-proxy"
Oct 30 03:12:25.244911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.244891    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.269371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.269223    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:12:25.271066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.271033    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f21058-serving-cert\""
Oct 30 03:12:25.278847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.278817    4214 generic.go:356] "Generic (PLEG): container finished" podID="844932dc-68d5-495b-9be3-b9023e0ba1be" containerID="5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7" exitCode=0
Oct 30 03:12:25.279005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.278879    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerDied","Data":"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"}
Oct 30 03:12:25.279005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.278889    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"
Oct 30 03:12:25.279005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.278924    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b" event={"ID":"844932dc-68d5-495b-9be3-b9023e0ba1be","Type":"ContainerDied","Data":"52e117dd91e39d0866efcc2d6f1ea8f2d791df10694f5ff1488507eb8a001e42"}
Oct 30 03:12:25.279005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.278951    4214 scope.go:122] "RemoveContainer" containerID="5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"
Oct 30 03:12:25.315981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.315946    4214 scope.go:122] "RemoveContainer" containerID="dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"
Oct 30 03:12:25.329345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.329298    4214 scope.go:122] "RemoveContainer" containerID="5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"
Oct 30 03:12:25.330154 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:25.329888    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7\": container with ID starting with 5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7 not found: ID does not exist" containerID="5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"
Oct 30 03:12:25.330154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.329975    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7"} err="failed to get container status \"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7\": rpc error: code = NotFound desc = could not find container \"5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7\": container with ID starting with 5f04e83749d4fec636cb5e4f1b9fa55c3c40bf09d51063b97f2fa5b6df31fdb7 not found: ID does not exist"
Oct 30 03:12:25.330154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.330016    4214 scope.go:122] "RemoveContainer" containerID="dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"
Oct 30 03:12:25.330441 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:25.330415    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c\": container with ID starting with dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c not found: ID does not exist" containerID="dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"
Oct 30 03:12:25.330511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.330441    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c"} err="failed to get container status \"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c\": rpc error: code = NotFound desc = could not find container \"dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c\": container with ID starting with dfa59db8c2a692db15a0a2433d5f7fdf2345826a7d748b5db5a2b6806f56514c not found: ID does not exist"
Oct 30 03:12:25.349647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.349585    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:12:25.356426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.356398    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-67440d-openshift-default-5584bd5bc-t5m4b"]
Oct 30 03:12:25.416787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.416757    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.416964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.416823    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.416964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.416926    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.417109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.417092    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwscv\" (UniqueName: \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.537255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.536719    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.537255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.536830    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.537255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.536941    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.548250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.545964    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.566505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.566431    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwscv\" (UniqueName: \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.570655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.570596    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.589252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.575718    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.631299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.625430    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwscv\" (UniqueName: \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\") pod \"maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:25.864757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:25.864712    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:26.081203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.081164    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:12:26.086847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.086817    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.099796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.099766    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:12:26.177649 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177522    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.177876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177669    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.177876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177791    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.177876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177829    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.178052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177918    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.178052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.177988    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.178192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.178172    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsdb\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.178311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.178293    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.178408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.178359    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.188950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.185520    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:12:26.191788 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:12:26.191760    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9650431d_405f_43c3_a7ca_198df859657d.slice/crio-b554e274840861f3461a620973b817cfce347e0e07c0ccd3fce662d6680fb251 WatchSource:0}: Error finding container b554e274840861f3461a620973b817cfce347e0e07c0ccd3fce662d6680fb251: Status 404 returned error can't find the container with id b554e274840861f3461a620973b817cfce347e0e07c0ccd3fce662d6680fb251
Oct 30 03:12:26.280482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280433    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280499    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280568    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280599    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280653    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280718    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.280897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280862    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsdb\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280898    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280978    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.280985    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.281022    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.281408    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.281533    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.281900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.281876    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.283579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.283553    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.283871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.283854    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.290562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.290535    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" event={"ID":"9650431d-405f-43c3-a7ca-198df859657d","Type":"ContainerStarted","Data":"b554e274840861f3461a620973b817cfce347e0e07c0ccd3fce662d6680fb251"}
Oct 30 03:12:26.301207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.301188    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsdb\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.301409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.301391    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.436351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.436069    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:26.877670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.877621    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:12:26.881183 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:12:26.881147    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91188950_a29c_4667_86a5_ed130774a70d.slice/crio-ded805fad56fba49e3609df741f34ea2eab92a3ca977fef3b388ab91f061797c WatchSource:0}: Error finding container ded805fad56fba49e3609df741f34ea2eab92a3ca977fef3b388ab91f061797c: Status 404 returned error can't find the container with id ded805fad56fba49e3609df741f34ea2eab92a3ca977fef3b388ab91f061797c
Oct 30 03:12:26.884717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.884679    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:26.884802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.884770    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:26.884842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:26.884815    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:12:27.075360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.075322    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844932dc-68d5-495b-9be3-b9023e0ba1be" path="/var/lib/kubelet/pods/844932dc-68d5-495b-9be3-b9023e0ba1be/volumes"
Oct 30 03:12:27.347251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.344566    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" event={"ID":"91188950-a29c-4667-86a5-ed130774a70d","Type":"ContainerStarted","Data":"71dc7a8f99ce9271f68db837da20ffd42d9bd60dd065ac43d9df49b71d275b5b"}
Oct 30 03:12:27.347251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.344659    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" event={"ID":"91188950-a29c-4667-86a5-ed130774a70d","Type":"ContainerStarted","Data":"ded805fad56fba49e3609df741f34ea2eab92a3ca977fef3b388ab91f061797c"}
Oct 30 03:12:27.350457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.349720    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" event={"ID":"9650431d-405f-43c3-a7ca-198df859657d","Type":"ContainerStarted","Data":"52afac48556bc83cfe80b6bbdfa473aee74af725def27b3481e3532c931aa601"}
Oct 30 03:12:27.356246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.351088    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:27.412341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.412275    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podStartSLOduration=1.41225889 podStartE2EDuration="1.41225889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:27.407577523 +0000 UTC m=+1886.481748316" watchObservedRunningTime="2026-10-30 03:12:27.41225889 +0000 UTC m=+1886.486429683"
Oct 30 03:12:27.444259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.438546    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:27.445542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.445512    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:27.445733 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.445705    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:27.455084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:27.455027    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" podStartSLOduration=1.9609096400000001 podStartE2EDuration="2.455012104s" totalImagesPullingTime="494.102464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:27.452278314 +0000 UTC m=+1886.526449106" watchObservedRunningTime="2026-10-30 03:12:27.455012104 +0000 UTC m=+1886.529182897"
Oct 30 03:12:28.445406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:28.445367    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:28.446008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:28.445973    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:28.866253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:28.861828    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:12:28.880863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:28.880829    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:12:29.467029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:29.457715    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:29.467029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:29.457775    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:30.445255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:30.443615    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:30.445255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:30.443676    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:31.445632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:31.445595    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:31.446203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:31.446169    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:32.264255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:32.258458    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:12:32.448275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:32.446538    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:32.448275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:32.446600    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:33.381689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:33.381659    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:12:33.446307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:33.442686    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 03:12:33.446307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:33.442767    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 03:12:33.862589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:33.862533    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" podUID="441fce81-461e-49b5-b46f-f7528bc005cb" containerName="payload-processing" containerID="cri-o://7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed" gracePeriod=25
Oct 30 03:12:33.885380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:33.885303    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" podUID="5dba0bb9-16dc-4158-aedf-bf4384558c61" containerName="payload-pre-processing" containerID="cri-o://e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059" gracePeriod=25
Oct 30 03:12:34.240267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.240219    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:12:34.307193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.307164    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:12:34.431183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.431085    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\") pod \"5dba0bb9-16dc-4158-aedf-bf4384558c61\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") "
Oct 30 03:12:34.431183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.431181    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\") pod \"441fce81-461e-49b5-b46f-f7528bc005cb\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") "
Oct 30 03:12:34.431465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.431249    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\" (UniqueName: \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\") pod \"5dba0bb9-16dc-4158-aedf-bf4384558c61\" (UID: \"5dba0bb9-16dc-4158-aedf-bf4384558c61\") "
Oct 30 03:12:34.431465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.431445    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\" (UniqueName: \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\") pod \"441fce81-461e-49b5-b46f-f7528bc005cb\" (UID: \"441fce81-461e-49b5-b46f-f7528bc005cb\") "
Oct 30 03:12:34.432941 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.432908    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume" pod "441fce81-461e-49b5-b46f-f7528bc005cb" (UID: "441fce81-461e-49b5-b46f-f7528bc005cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:34.434187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.434037    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume" pod "5dba0bb9-16dc-4158-aedf-bf4384558c61" (UID: "5dba0bb9-16dc-4158-aedf-bf4384558c61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:34.444494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.437498    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc" pod "5dba0bb9-16dc-4158-aedf-bf4384558c61" (UID: "5dba0bb9-16dc-4158-aedf-bf4384558c61"). InnerVolumeSpecName "kube-api-access-hmsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:34.452319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.450749    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4" pod "441fce81-461e-49b5-b46f-f7528bc005cb" (UID: "441fce81-461e-49b5-b46f-f7528bc005cb"). InnerVolumeSpecName "kube-api-access-v26r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:34.510353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.510294    4214 generic.go:356] "Generic (PLEG): container finished" podID="441fce81-461e-49b5-b46f-f7528bc005cb" containerID="7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed" exitCode=0
Oct 30 03:12:34.510555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.510388    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" event={"ID":"441fce81-461e-49b5-b46f-f7528bc005cb","Type":"ContainerDied","Data":"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"}
Oct 30 03:12:34.510555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.510443    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp" event={"ID":"441fce81-461e-49b5-b46f-f7528bc005cb","Type":"ContainerDied","Data":"01430bcf130bda97abb7bdf28256e2ab5e6c42960a11f57fbbdbac68cf5ca0cb"}
Oct 30 03:12:34.510555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.510472    4214 scope.go:122] "RemoveContainer" containerID="7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"
Oct 30 03:12:34.510733 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.510636    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"
Oct 30 03:12:34.514825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.514694    4214 generic.go:356] "Generic (PLEG): container finished" podID="5dba0bb9-16dc-4158-aedf-bf4384558c61" containerID="e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059" exitCode=0
Oct 30 03:12:34.514980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.514906    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" event={"ID":"5dba0bb9-16dc-4158-aedf-bf4384558c61","Type":"ContainerDied","Data":"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"}
Oct 30 03:12:34.515062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.514988    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k" event={"ID":"5dba0bb9-16dc-4158-aedf-bf4384558c61","Type":"ContainerDied","Data":"b57a88a0f88f645be46614e109841453ab27b7098677e978476e9a4937bb0386"}
Oct 30 03:12:34.515062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.515058    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"
Oct 30 03:12:34.537968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.536365    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dba0bb9-16dc-4158-aedf-bf4384558c61-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:34.537968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.536393    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441fce81-461e-49b5-b46f-f7528bc005cb-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:34.537968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.536408    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsvc\" (UniqueName: \"kubernetes.io/projected/5dba0bb9-16dc-4158-aedf-bf4384558c61-kube-api-access-hmsvc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:34.537968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.536422    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26r4\" (UniqueName: \"kubernetes.io/projected/441fce81-461e-49b5-b46f-f7528bc005cb-kube-api-access-v26r4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:34.567902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.567882    4214 scope.go:122] "RemoveContainer" containerID="7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"
Oct 30 03:12:34.574139 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:34.574105    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed\": container with ID starting with 7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed not found: ID does not exist" containerID="7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"
Oct 30 03:12:34.574336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.574149    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed"} err="failed to get container status \"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed\": rpc error: code = NotFound desc = could not find container \"7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed\": container with ID starting with 7e891c31face94068ffe0371ac5d7c02fc5a4ea411ba91e9924cbc5c4b1881ed not found: ID does not exist"
Oct 30 03:12:34.574336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.574185    4214 scope.go:122] "RemoveContainer" containerID="e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"
Oct 30 03:12:34.605472 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.605438    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:12:34.610439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.610409    4214 scope.go:122] "RemoveContainer" containerID="e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"
Oct 30 03:12:34.610806 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:34.610781    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059\": container with ID starting with e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059 not found: ID does not exist" containerID="e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"
Oct 30 03:12:34.610896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.610816    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059"} err="failed to get container status \"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059\": rpc error: code = NotFound desc = could not find container \"e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059\": container with ID starting with e7043148d77fde37d8a8bed2f3ac35fb70975efd6b5d5120ce6bce908be33059 not found: ID does not exist"
Oct 30 03:12:34.614436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.614401    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-734d43-f88dd95f4-tsstp"]
Oct 30 03:12:34.633523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.633492    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:12:34.641964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.641921    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-734d43-67ff74f87f-fzb8k"]
Oct 30 03:12:34.753536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.753456    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:34.754529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.754502    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:34.755364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:34.755341    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:12:35.071876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:35.071841    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441fce81-461e-49b5-b46f-f7528bc005cb" path="/var/lib/kubelet/pods/441fce81-461e-49b5-b46f-f7528bc005cb/volumes"
Oct 30 03:12:35.072356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:35.072340    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dba0bb9-16dc-4158-aedf-bf4384558c61" path="/var/lib/kubelet/pods/5dba0bb9-16dc-4158-aedf-bf4384558c61/volumes"
Oct 30 03:12:37.610955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.610919    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:12:37.612039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612011    4214 cpu_manager.go:414] "containerMap: removing container" podUID="441fce81-461e-49b5-b46f-f7528bc005cb" containerName="payload-processing"
Oct 30 03:12:37.612039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612038    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441fce81-461e-49b5-b46f-f7528bc005cb" containerName="payload-processing"
Oct 30 03:12:37.612183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612095    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5dba0bb9-16dc-4158-aedf-bf4384558c61" containerName="payload-pre-processing"
Oct 30 03:12:37.612183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612103    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dba0bb9-16dc-4158-aedf-bf4384558c61" containerName="payload-pre-processing"
Oct 30 03:12:37.612366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612350    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dba0bb9-16dc-4158-aedf-bf4384558c61" containerName="payload-pre-processing"
Oct 30 03:12:37.612407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.612382    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="441fce81-461e-49b5-b46f-f7528bc005cb" containerName="payload-processing"
Oct 30 03:12:37.618805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.618777    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.641276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.640736    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-537001\"/\"openshift-service-ca.crt\""
Oct 30 03:12:37.641276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.640949    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-537001\"/\"default-dockercfg-pnzvl\""
Oct 30 03:12:37.643150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.643127    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:12:37.651983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.651958    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-537001\"/\"test-model-537001-kserve-self-signed-certs\""
Oct 30 03:12:37.651983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.651971    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-537001\"/\"kube-root-ca.crt\""
Oct 30 03:12:37.700690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.700657    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.700867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.700699    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.700867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.700763    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.700961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.700892    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.701016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.701002    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.701117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.701027    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2kj\" (UniqueName: \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803077    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803120    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803151    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803255    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803420 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803338    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803420 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803358    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2kj\" (UniqueName: \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803602    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803729    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.803839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.803737    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.805350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.805331    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.806053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.806026    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.823844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.823816    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2kj\" (UniqueName: \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\") pod \"test-model-537001-kserve-778c6f658c-8p4jp\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") " pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:37.944302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:37.942540    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:38.420400 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:12:38.420371    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c6a2d0a_b5c2_4d95_ae03_e5bed8c53509.slice/crio-d26629a807e2b37b018f844b2e40a6669888f9bf8dbf1ce444ec4753a636f14a WatchSource:0}: Error finding container d26629a807e2b37b018f844b2e40a6669888f9bf8dbf1ce444ec4753a636f14a: Status 404 returned error can't find the container with id d26629a807e2b37b018f844b2e40a6669888f9bf8dbf1ce444ec4753a636f14a
Oct 30 03:12:38.420892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:38.420866    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:12:38.580253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:38.577645    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" event={"ID":"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509","Type":"ContainerStarted","Data":"d8888dbf0acfa4ad29122b6750d4cef5af1d5f4ae3e457e8e600c1b3336dc864"}
Oct 30 03:12:38.580253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:38.577699    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" event={"ID":"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509","Type":"ContainerStarted","Data":"d26629a807e2b37b018f844b2e40a6669888f9bf8dbf1ce444ec4753a636f14a"}
Oct 30 03:12:38.645253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:38.634085    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podStartSLOduration=1.634066204 podStartE2EDuration="1.634066204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:38.623581618 +0000 UTC m=+1897.697752411" watchObservedRunningTime="2026-10-30 03:12:38.634066204 +0000 UTC m=+1897.708237002"
Oct 30 03:12:38.943138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:38.943099    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:39.173191 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:39.173151    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:12:43.012302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.011370    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:12:43.031516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.031486    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.035046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.035017    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:12:43.048523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.048500    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1fce3\"/\"openshift-service-ca.crt\""
Oct 30 03:12:43.048800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.048774    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1fce3\"/\"kube-root-ca.crt\""
Oct 30 03:12:43.067086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.067055    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1fce3\"/\"test-model-c1fce3-kserve-self-signed-certs\""
Oct 30 03:12:43.067288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.067172    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1fce3\"/\"default-dockercfg-zjsgt\""
Oct 30 03:12:43.151352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151315    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.151543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151394    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.151543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151427    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.151631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151552    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.151687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151636    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.151823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.151805    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srk68\" (UniqueName: \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.274985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.274864    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.275166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.275045    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.275261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.275221    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srk68\" (UniqueName: \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.276108    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.276261    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.276294    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.276340    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.276877    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.283083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.277099    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.330250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.325032    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srk68\" (UniqueName: \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.330250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.325803    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.330250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.328421    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\") pod \"test-model-c1fce3-kserve-7c6c7b94bf-lb5z4\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") " pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.365252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.355025    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:43.516603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.516195    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:12:43.516603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.516445    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" podUID="50970dbb-e48e-4275-8bee-78c0d97bd665" containerName="maas-api" containerID="cri-o://5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7" gracePeriod=30
Oct 30 03:12:43.756007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.754430    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:12:43.758591 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:12:43.758552    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2139de_66e7_4906_8dfb_35015be69017.slice/crio-09e0240ece739eb4cddc4ac195a8603125e83ec705d07dfe6c53603dfcfe3df4 WatchSource:0}: Error finding container 09e0240ece739eb4cddc4ac195a8603125e83ec705d07dfe6c53603dfcfe3df4: Status 404 returned error can't find the container with id 09e0240ece739eb4cddc4ac195a8603125e83ec705d07dfe6c53603dfcfe3df4
Oct 30 03:12:43.843569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:43.843542    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:12:44.027470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.014217    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\") pod \"50970dbb-e48e-4275-8bee-78c0d97bd665\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") "
Oct 30 03:12:44.043963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.043924    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\") pod \"50970dbb-e48e-4275-8bee-78c0d97bd665\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") "
Oct 30 03:12:44.074250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.074151    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\") pod \"50970dbb-e48e-4275-8bee-78c0d97bd665\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") "
Oct 30 03:12:44.074517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.074497    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\" (UniqueName: \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\") pod \"50970dbb-e48e-4275-8bee-78c0d97bd665\" (UID: \"50970dbb-e48e-4275-8bee-78c0d97bd665\") "
Oct 30 03:12:44.103402 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.048495    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca" pod "50970dbb-e48e-4275-8bee-78c0d97bd665" (UID: "50970dbb-e48e-4275-8bee-78c0d97bd665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:44.234394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.234325    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50970dbb-e48e-4275-8bee-78c0d97bd665-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.361587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.361463    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs" pod "50970dbb-e48e-4275-8bee-78c0d97bd665" (UID: "50970dbb-e48e-4275-8bee-78c0d97bd665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:44.361587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.361557    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls" pod "50970dbb-e48e-4275-8bee-78c0d97bd665" (UID: "50970dbb-e48e-4275-8bee-78c0d97bd665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:44.362011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.361986    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr" pod "50970dbb-e48e-4275-8bee-78c0d97bd665" (UID: "50970dbb-e48e-4275-8bee-78c0d97bd665"). InnerVolumeSpecName "kube-api-access-shkvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:44.439089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.439010    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.439089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.439046    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50970dbb-e48e-4275-8bee-78c0d97bd665-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.439089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.439059    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkvr\" (UniqueName: \"kubernetes.io/projected/50970dbb-e48e-4275-8bee-78c0d97bd665-kube-api-access-shkvr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.756416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.755622    4214 generic.go:356] "Generic (PLEG): container finished" podID="50970dbb-e48e-4275-8bee-78c0d97bd665" containerID="5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7" exitCode=0
Oct 30 03:12:44.756416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.755975    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" event={"ID":"50970dbb-e48e-4275-8bee-78c0d97bd665","Type":"ContainerDied","Data":"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"}
Oct 30 03:12:44.756416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.756035    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q" event={"ID":"50970dbb-e48e-4275-8bee-78c0d97bd665","Type":"ContainerDied","Data":"d698ef5fc75cb7b3e341fd3650e0669e56f0570a9c1593296281e9f35c66f8d1"}
Oct 30 03:12:44.756416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.756064    4214 scope.go:122] "RemoveContainer" containerID="5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"
Oct 30 03:12:44.768771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.768734    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" event={"ID":"6f2139de-66e7-4906-8dfb-35015be69017","Type":"ContainerStarted","Data":"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"}
Oct 30 03:12:44.769246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.769178    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" event={"ID":"6f2139de-66e7-4906-8dfb-35015be69017","Type":"ContainerStarted","Data":"09e0240ece739eb4cddc4ac195a8603125e83ec705d07dfe6c53603dfcfe3df4"}
Oct 30 03:12:44.772475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.772453    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"
Oct 30 03:12:44.816876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.816785    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" podStartSLOduration=2.816765906 podStartE2EDuration="2.816765906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:44.808560539 +0000 UTC m=+1903.882731334" watchObservedRunningTime="2026-10-30 03:12:44.816765906 +0000 UTC m=+1903.890936700"
Oct 30 03:12:44.829021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.828995    4214 scope.go:122] "RemoveContainer" containerID="5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"
Oct 30 03:12:44.830757 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:12:44.830722    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7\": container with ID starting with 5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7 not found: ID does not exist" containerID="5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"
Oct 30 03:12:44.830879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:44.830766    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7"} err="failed to get container status \"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7\": rpc error: code = NotFound desc = could not find container \"5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7\": container with ID starting with 5d24989abfdc945e250ca9494ed778c0d11a842d1aadc1fe62c2353af64aeea7 not found: ID does not exist"
Oct 30 03:12:45.561274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:45.559367    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:45.561274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:45.559445    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:12:45.597447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:45.597396    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d30e9f-c64869b76-nf82q"]
Oct 30 03:12:47.082267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:47.082210    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50970dbb-e48e-4275-8bee-78c0d97bd665" path="/var/lib/kubelet/pods/50970dbb-e48e-4275-8bee-78c0d97bd665/volumes"
Oct 30 03:12:47.947556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:47.945741    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:47.961458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:47.961430    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:48.820639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:48.820614    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:12:53.355881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:53.355844    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:53.373277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:53.373116    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:12:53.899483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:12:53.899446    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:13:03.338337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:03.338297    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:13:03.372214 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:03.372176    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:13:06.153367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:06.153331    4214 scope.go:122] "RemoveContainer" containerID="f57b3b4f031548c0a48d2aa9dce389e9ef66130a25b4fb8f9286b0ab54ff99f0"
Oct 30 03:13:08.339506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:08.339467    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing" containerID="cri-o://9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae" gracePeriod=25
Oct 30 03:13:08.373464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:08.373423    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing" containerID="cri-o://91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d" gracePeriod=25
Oct 30 03:13:08.511603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:08.511558    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:13:09.168071 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:09.168029    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:13:09.519811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:09.519720    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:13:15.177552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.177350    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:13:15.179410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.179381    4214 cpu_manager.go:414] "containerMap: removing container" podUID="50970dbb-e48e-4275-8bee-78c0d97bd665" containerName="maas-api"
Oct 30 03:13:15.179562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.179549    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50970dbb-e48e-4275-8bee-78c0d97bd665" containerName="maas-api"
Oct 30 03:13:15.179981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.179964    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="50970dbb-e48e-4275-8bee-78c0d97bd665" containerName="maas-api"
Oct 30 03:13:15.185818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.185792    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.204549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.204522    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2305f4-openshift-default-dockercfg-l7spl\""
Oct 30 03:13:15.208456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.208428    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:13:15.310703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.310664    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.310883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.310863    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmtt\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311029    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311056    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311077    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311096    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311214 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311190    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311222    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.311271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.311248    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.412828    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.412881    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.412911    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.412940    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.413024    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.413064    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.413088    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.413174    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.413384    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmtt\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.414248    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.417290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.414547    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.443339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.415219    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.443339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.437867    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.447265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.444747    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.465177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.465103    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.476849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.476818    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.508415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.507648    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.563788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.563163    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmtt\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:15.810901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:15.810869    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:16.144878 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:13:16.144843    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dc1e3d_6a59_45ec_adaa_e951c3f3ced7.slice/crio-e68d0d43008508c3d228028a7aab0d8aa42e58d13b9034d2e61a51d54e07e799 WatchSource:0}: Error finding container e68d0d43008508c3d228028a7aab0d8aa42e58d13b9034d2e61a51d54e07e799: Status 404 returned error can't find the container with id e68d0d43008508c3d228028a7aab0d8aa42e58d13b9034d2e61a51d54e07e799
Oct 30 03:13:16.147295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.147265    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:13:16.148207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.148178    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:16.148281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.148261    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:16.148317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.148291    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:16.211539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.211508    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerStarted","Data":"e68d0d43008508c3d228028a7aab0d8aa42e58d13b9034d2e61a51d54e07e799"}
Oct 30 03:13:16.458111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.458012    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:13:16.458349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.458291    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy" containerID="cri-o://85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b" gracePeriod=30
Oct 30 03:13:16.474830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.473800    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:13:16.474830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:16.474033    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy" containerID="cri-o://9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f" gracePeriod=30
Oct 30 03:13:17.223312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:17.223267    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerStarted","Data":"c770c796ace441054a6dbd444f2baee2ef64a7e359a1a157d0aedec409b8211c"}
Oct 30 03:13:17.284154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:17.284098    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podStartSLOduration=2.284080578 podStartE2EDuration="2.284080578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:17.279408517 +0000 UTC m=+1936.353579309" watchObservedRunningTime="2026-10-30 03:13:17.284080578 +0000 UTC m=+1936.358251371"
Oct 30 03:13:17.811146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:17.811114    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:17.817199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:17.817169    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:18.250021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:18.249921    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:18.251425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:18.251397    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:13:18.511442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:18.511360    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:13:19.520779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:19.520739    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:13:22.226828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.226748    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:13:22.297526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.297498    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:13:22.301350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.301325    4214 generic.go:356] "Generic (PLEG): container finished" podID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerID="85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b" exitCode=0
Oct 30 03:13:22.301509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.301377    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"
Oct 30 03:13:22.301509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.301411    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" event={"ID":"2a26e7b8-42fe-408d-b5bd-f22332f208c4","Type":"ContainerDied","Data":"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"}
Oct 30 03:13:22.301509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.301500    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl" event={"ID":"2a26e7b8-42fe-408d-b5bd-f22332f208c4","Type":"ContainerDied","Data":"26c849d87240794c8a08d759f22c6226d1af80d49ca8abda1116ba916a752e67"}
Oct 30 03:13:22.301677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.301532    4214 scope.go:122] "RemoveContainer" containerID="85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"
Oct 30 03:13:22.303057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.303033    4214 generic.go:356] "Generic (PLEG): container finished" podID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerID="9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f" exitCode=0
Oct 30 03:13:22.303183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.303088    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"
Oct 30 03:13:22.303183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.303122    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" event={"ID":"26fa08d2-b0c5-4e69-84b9-590f404b6540","Type":"ContainerDied","Data":"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"}
Oct 30 03:13:22.303183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.303166    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s" event={"ID":"26fa08d2-b0c5-4e69-84b9-590f404b6540","Type":"ContainerDied","Data":"bf9b0a9925271b0887a90db7dbddf720ac2d1ae0baa60ad9d081513a80012a22"}
Oct 30 03:13:22.313761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.313729    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.313913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.313825    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.313913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.313871    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.313919    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314011    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314047    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314088    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314118    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314149    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314179    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314215    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314278    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314317    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314364    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314481    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\") pod \"26fa08d2-b0c5-4e69-84b9-590f404b6540\" (UID: \"26fa08d2-b0c5-4e69-84b9-590f404b6540\") "
Oct 30 03:13:22.314789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314637    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314674    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.314789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.314731    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\") pod \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\" (UID: \"2a26e7b8-42fe-408d-b5bd-f22332f208c4\") "
Oct 30 03:13:22.315810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.315687    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.315810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.315731    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.316588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.316562    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.316886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.316858    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:22.317176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.317151    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.317455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.317429    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.318483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.318271    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.319211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.318564    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.319582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.319402    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.319883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.319725    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.320118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320051    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:22.320450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320410    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320441    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fa08d2-b0c5-4e69-84b9-590f404b6540-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320480    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320492    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320505    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320516    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320528    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320542    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320588    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.320604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320604    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.321042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.320662    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a26e7b8-42fe-408d-b5bd-f22332f208c4-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.322723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.322694    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:13:22.322929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.322905    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:22.323790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323375    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:13:22.323790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323401    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:22.323790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323431    4214 scope.go:122] "RemoveContainer" containerID="85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"
Oct 30 03:13:22.323790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323440    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:22.323790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323489    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g" pod "2a26e7b8-42fe-408d-b5bd-f22332f208c4" (UID: "2a26e7b8-42fe-408d-b5bd-f22332f208c4"). InnerVolumeSpecName "kube-api-access-td98g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:22.324113 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:22.323886    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b\": container with ID starting with 85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b not found: ID does not exist" containerID="85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"
Oct 30 03:13:22.324113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323925    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b"} err="failed to get container status \"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b\": rpc error: code = NotFound desc = could not find container \"85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b\": container with ID starting with 85edefbf7380ff3c0a1ec5bd83a1f9a9131bd2a82cb473670a7469ca11c17e5b not found: ID does not exist"
Oct 30 03:13:22.324113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.323950    4214 scope.go:122] "RemoveContainer" containerID="9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"
Oct 30 03:13:22.326955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.326926    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc" pod "26fa08d2-b0c5-4e69-84b9-590f404b6540" (UID: "26fa08d2-b0c5-4e69-84b9-590f404b6540"). InnerVolumeSpecName "kube-api-access-n69zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:22.336377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.336354    4214 scope.go:122] "RemoveContainer" containerID="9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"
Oct 30 03:13:22.336720 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:22.336690    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f\": container with ID starting with 9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f not found: ID does not exist" containerID="9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"
Oct 30 03:13:22.336799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.336730    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f"} err="failed to get container status \"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f\": rpc error: code = NotFound desc = could not find container \"9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f\": container with ID starting with 9966692f82bd122c6f5d911f301dc75c1b0e5142d494d003139a38ecc34bdd6f not found: ID does not exist"
Oct 30 03:13:22.421852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421807    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.421852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421839    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td98g\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-kube-api-access-td98g\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.421852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421858    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69zc\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-kube-api-access-n69zc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.422181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421868    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.422181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421879    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.422181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421887    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fa08d2-b0c5-4e69-84b9-590f404b6540-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.422181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.421894    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a26e7b8-42fe-408d-b5bd-f22332f208c4-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:22.669662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.669613    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:13:22.675598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.675569    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6h2wl"]
Oct 30 03:13:22.711669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.711632    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:13:22.716833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:22.716809    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kpx2s"]
Oct 30 03:13:23.074133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:23.074093    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" path="/var/lib/kubelet/pods/26fa08d2-b0c5-4e69-84b9-590f404b6540/volumes"
Oct 30 03:13:23.074879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:23.074856    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" path="/var/lib/kubelet/pods/2a26e7b8-42fe-408d-b5bd-f22332f208c4/volumes"
Oct 30 03:13:28.515518 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:13:28.510110    4214 logging.go:55] [core] [Channel #6734 SubChannel #6735]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.178:50052", ServerName: "10.133.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.178:50052: connect: connection refused"
Oct 30 03:13:28.957823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:28.957428    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:13:29.061254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.056768    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:13:29.120252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.119708    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume" pod "9577a1a5-4200-41e9-a6ec-5274f09990aa" (UID: "9577a1a5-4200-41e9-a6ec-5274f09990aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:29.120252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.119813    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\") pod \"9577a1a5-4200-41e9-a6ec-5274f09990aa\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") "
Oct 30 03:13:29.120485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.120282    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\" (UniqueName: \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\") pod \"9577a1a5-4200-41e9-a6ec-5274f09990aa\" (UID: \"9577a1a5-4200-41e9-a6ec-5274f09990aa\") "
Oct 30 03:13:29.164310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.152376    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9577a1a5-4200-41e9-a6ec-5274f09990aa-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:29.280265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.272009    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\") pod \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") "
Oct 30 03:13:29.368707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.368668    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") pod \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") "
Oct 30 03:13:29.396826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.289507    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume" pod "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" (UID: "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:29.648304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.550464    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f" pod "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" (UID: "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5"). InnerVolumeSpecName "kube-api-access-6nn7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:29.656313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.541599    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.178:50052\" within 1s: context deadline exceeded"
Oct 30 03:13:29.723256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.713775    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") pod \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\" (UID: \"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5\") "
Oct 30 03:13:29.723256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.714822    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:29.723256 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:13:29.714941    4214 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5/volumes/kubernetes.io~projected/kube-api-access-6nn7f
Oct 30 03:13:29.723256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.715271    4214 generic.go:356] "Generic (PLEG): container finished" podID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerID="91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d" exitCode=0
Oct 30 03:13:29.723256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.715487    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:13:29.725291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.725258    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn" pod "9577a1a5-4200-41e9-a6ec-5274f09990aa" (UID: "9577a1a5-4200-41e9-a6ec-5274f09990aa"). InnerVolumeSpecName "kube-api-access-q5thn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:29.725820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.725796    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f" pod "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" (UID: "d7763d57-b2bd-4d42-8d64-0f9c47bf89f5"). InnerVolumeSpecName "kube-api-access-6nn7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:29.789902 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.789869    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"
Oct 30 03:13:29.933252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.860387    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" event={"ID":"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5","Type":"ContainerDied","Data":"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"}
Oct 30 03:13:29.933252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.860453    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm" event={"ID":"d7763d57-b2bd-4d42-8d64-0f9c47bf89f5","Type":"ContainerDied","Data":"e9d3063d66665545abb014d06df3aa43dbe23c63234d3ea5f762202ce22cea62"}
Oct 30 03:13:29.933252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.860503    4214 scope.go:122] "RemoveContainer" containerID="91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"
Oct 30 03:13:29.933252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.830316    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5thn\" (UniqueName: \"kubernetes.io/projected/9577a1a5-4200-41e9-a6ec-5274f09990aa-kube-api-access-q5thn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:29.933252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:29.860699    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nn7f\" (UniqueName: \"kubernetes.io/projected/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5-kube-api-access-6nn7f\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:30.121995 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.121958    4214 generic.go:356] "Generic (PLEG): container finished" podID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerID="9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae" exitCode=0
Oct 30 03:13:30.122145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.122118    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" event={"ID":"9577a1a5-4200-41e9-a6ec-5274f09990aa","Type":"ContainerDied","Data":"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"}
Oct 30 03:13:30.122205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.122178    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb" event={"ID":"9577a1a5-4200-41e9-a6ec-5274f09990aa","Type":"ContainerDied","Data":"1b5c1c90003b5af863584575b1557e5b44082145b3dd3f4c13e1b5258684648e"}
Oct 30 03:13:30.123955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.123928    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"
Oct 30 03:13:30.181257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.179803    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:13:30.221536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.221493    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cea203-5779bf5c55-d82pm"]
Oct 30 03:13:30.229834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.229803    4214 scope.go:122] "RemoveContainer" containerID="91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"
Oct 30 03:13:30.230155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.230139    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:13:30.230528 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:30.230462    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d\": container with ID starting with 91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d not found: ID does not exist" containerID="91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"
Oct 30 03:13:30.230633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.230541    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d"} err="failed to get container status \"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d\": rpc error: code = NotFound desc = could not find container \"91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d\": container with ID starting with 91306991fdc7e6ef19c3721fa4b7f6eafe01d8736fbde8b1dd332fdead51d92d not found: ID does not exist"
Oct 30 03:13:30.230633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.230568    4214 scope.go:122] "RemoveContainer" containerID="9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"
Oct 30 03:13:30.289721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.289679    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cea203-f4cc9fbcd-bx8rb"]
Oct 30 03:13:30.299813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.299787    4214 scope.go:122] "RemoveContainer" containerID="9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"
Oct 30 03:13:30.300282 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:30.300186    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae\": container with ID starting with 9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae not found: ID does not exist" containerID="9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"
Oct 30 03:13:30.300282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.300215    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae"} err="failed to get container status \"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae\": rpc error: code = NotFound desc = could not find container \"9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae\": container with ID starting with 9f1b0fffdb48f2d85877314dc3fdf1bd30149f2a2ee6a3576a6105674cf448ae not found: ID does not exist"
Oct 30 03:13:30.899322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.899287    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:13:30.900759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:30.900473    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" podUID="b6960882-4dc2-45b0-9257-ec45a4714ce5" containerName="maas-api" containerID="cri-o://d67527f1e2738b612dfd4a452f87b00734c3fabdda3b7e801f4dae1f7100c54d" gracePeriod=30
Oct 30 03:13:31.196211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.196174    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" path="/var/lib/kubelet/pods/9577a1a5-4200-41e9-a6ec-5274f09990aa/volumes"
Oct 30 03:13:31.204694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.204662    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" path="/var/lib/kubelet/pods/d7763d57-b2bd-4d42-8d64-0f9c47bf89f5/volumes"
Oct 30 03:13:31.209600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.209577    4214 generic.go:356] "Generic (PLEG): container finished" podID="b6960882-4dc2-45b0-9257-ec45a4714ce5" containerID="d67527f1e2738b612dfd4a452f87b00734c3fabdda3b7e801f4dae1f7100c54d" exitCode=0
Oct 30 03:13:31.209806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.209791    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" event={"ID":"b6960882-4dc2-45b0-9257-ec45a4714ce5","Type":"ContainerDied","Data":"d67527f1e2738b612dfd4a452f87b00734c3fabdda3b7e801f4dae1f7100c54d"}
Oct 30 03:13:31.889677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.889334    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:13:31.977326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.977287    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\") pod \"b6960882-4dc2-45b0-9257-ec45a4714ce5\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") "
Oct 30 03:13:31.977811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.977450    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\" (UniqueName: \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\") pod \"b6960882-4dc2-45b0-9257-ec45a4714ce5\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") "
Oct 30 03:13:31.977811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.977572    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\") pod \"b6960882-4dc2-45b0-9257-ec45a4714ce5\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") "
Oct 30 03:13:31.977934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.977914    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\") pod \"b6960882-4dc2-45b0-9257-ec45a4714ce5\" (UID: \"b6960882-4dc2-45b0-9257-ec45a4714ce5\") "
Oct 30 03:13:31.979338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.979301    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca" pod "b6960882-4dc2-45b0-9257-ec45a4714ce5" (UID: "b6960882-4dc2-45b0-9257-ec45a4714ce5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:31.980771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.980742    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs" pod "b6960882-4dc2-45b0-9257-ec45a4714ce5" (UID: "b6960882-4dc2-45b0-9257-ec45a4714ce5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:31.983329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.983295    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls" pod "b6960882-4dc2-45b0-9257-ec45a4714ce5" (UID: "b6960882-4dc2-45b0-9257-ec45a4714ce5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:31.985312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:31.985220    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px" pod "b6960882-4dc2-45b0-9257-ec45a4714ce5" (UID: "b6960882-4dc2-45b0-9257-ec45a4714ce5"). InnerVolumeSpecName "kube-api-access-7q9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:32.092313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.088789    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9px\" (UniqueName: \"kubernetes.io/projected/b6960882-4dc2-45b0-9257-ec45a4714ce5-kube-api-access-7q9px\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:32.092313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.088903    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:32.092313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.088918    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6960882-4dc2-45b0-9257-ec45a4714ce5-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:32.092313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.088931    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6960882-4dc2-45b0-9257-ec45a4714ce5-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:32.294154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.294115    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf" event={"ID":"b6960882-4dc2-45b0-9257-ec45a4714ce5","Type":"ContainerDied","Data":"dc00f55c6b367be12c5d097265d3b41e64cc81fc4d8b109626b4620cb2ef95f1"}
Oct 30 03:13:32.294480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.294456    4214 scope.go:122] "RemoveContainer" containerID="d67527f1e2738b612dfd4a452f87b00734c3fabdda3b7e801f4dae1f7100c54d"
Oct 30 03:13:32.296181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.296156    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"
Oct 30 03:13:32.412540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.412507    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:13:32.417354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:32.417328    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cea203-5d7c9d7444-dl7bf"]
Oct 30 03:13:33.082835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:33.082798    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6960882-4dc2-45b0-9257-ec45a4714ce5" path="/var/lib/kubelet/pods/b6960882-4dc2-45b0-9257-ec45a4714ce5/volumes"
Oct 30 03:13:34.287531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.287489    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8809-a1fcaf"]
Oct 30 03:13:34.289425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.289396    4214 cpu_manager.go:414] "containerMap: removing container" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy"
Oct 30 03:13:34.289606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.289589    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy"
Oct 30 03:13:34.289728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.289715    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing"
Oct 30 03:13:34.289820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.289809    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing"
Oct 30 03:13:34.289942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.289930    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing"
Oct 30 03:13:34.290030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290018    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing"
Oct 30 03:13:34.290167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290155    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy"
Oct 30 03:13:34.290289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290275    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy"
Oct 30 03:13:34.290428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290416    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b6960882-4dc2-45b0-9257-ec45a4714ce5" containerName="maas-api"
Oct 30 03:13:34.290522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290511    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6960882-4dc2-45b0-9257-ec45a4714ce5" containerName="maas-api"
Oct 30 03:13:34.290797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290780    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6960882-4dc2-45b0-9257-ec45a4714ce5" containerName="maas-api"
Oct 30 03:13:34.290914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.290902    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fa08d2-b0c5-4e69-84b9-590f404b6540" containerName="istio-proxy"
Oct 30 03:13:34.291020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.291007    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9577a1a5-4200-41e9-a6ec-5274f09990aa" containerName="payload-processing"
Oct 30 03:13:34.291134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.291121    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7763d57-b2bd-4d42-8d64-0f9c47bf89f5" containerName="payload-pre-processing"
Oct 30 03:13:34.291270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.291242    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a26e7b8-42fe-408d-b5bd-f22332f208c4" containerName="istio-proxy"
Oct 30 03:13:34.298775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.298754    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:34.309839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.309804    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-a1fcaf"]
Oct 30 03:13:34.318493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.318272    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 03:13:34.477546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.477516    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmq6v\" (UniqueName: \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\") pod \"test-curl-8809-a1fcaf\" (UID: \"0cbe158e-11af-4307-a147-7bc46b788f50\") " pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:34.578832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.578796    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmq6v\" (UniqueName: \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\") pod \"test-curl-8809-a1fcaf\" (UID: \"0cbe158e-11af-4307-a147-7bc46b788f50\") " pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:34.600954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.600926    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmq6v\" (UniqueName: \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\") pod \"test-curl-8809-a1fcaf\" (UID: \"0cbe158e-11af-4307-a147-7bc46b788f50\") " pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:34.619466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.619437    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:34.975913 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:13:34.975519    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cbe158e_11af_4307_a147_7bc46b788f50.slice/crio-7d8c3d803cab3a4d3574741e57cd0dc70988de30498da28d02bc119793a49bf2 WatchSource:0}: Error finding container 7d8c3d803cab3a4d3574741e57cd0dc70988de30498da28d02bc119793a49bf2: Status 404 returned error can't find the container with id 7d8c3d803cab3a4d3574741e57cd0dc70988de30498da28d02bc119793a49bf2
Oct 30 03:13:34.978961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:34.978824    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8809-a1fcaf"]
Oct 30 03:13:35.328129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:35.328089    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-a1fcaf" event={"ID":"0cbe158e-11af-4307-a147-7bc46b788f50","Type":"ContainerStarted","Data":"2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526"}
Oct 30 03:13:35.328708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:35.328141    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8809-a1fcaf" event={"ID":"0cbe158e-11af-4307-a147-7bc46b788f50","Type":"ContainerStarted","Data":"7d8c3d803cab3a4d3574741e57cd0dc70988de30498da28d02bc119793a49bf2"}
Oct 30 03:13:35.366962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:35.366898    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8809-a1fcaf" podStartSLOduration=1.366880579 podStartE2EDuration="1.366880579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:35.365892049 +0000 UTC m=+1954.440062834" watchObservedRunningTime="2026-10-30 03:13:35.366880579 +0000 UTC m=+1954.441051375"
Oct 30 03:13:36.188936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:36.188890    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8809-a1fcaf"]
Oct 30 03:13:36.195693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:36.195662    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8809-a1fcaf"]
Oct 30 03:13:37.370087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:37.370042    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8809-a1fcaf" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" containerName="test-curl-8809-a1fcaf" containerID="cri-o://2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526" gracePeriod=2
Oct 30 03:13:38.729965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:38.729929    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:13:38.730385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:38.730187    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy" containerID="cri-o://1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a" gracePeriod=30
Oct 30 03:13:39.257207 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:39.257159    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:13:39.686434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:39.686408    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:39.744456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:39.744410    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\" (UniqueName: \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\") pod \"0cbe158e-11af-4307-a147-7bc46b788f50\" (UID: \"0cbe158e-11af-4307-a147-7bc46b788f50\") "
Oct 30 03:13:39.746719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:39.746687    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v" pod "0cbe158e-11af-4307-a147-7bc46b788f50" (UID: "0cbe158e-11af-4307-a147-7bc46b788f50"). InnerVolumeSpecName "kube-api-access-kmq6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:39.849701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:39.849656    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmq6v\" (UniqueName: \"kubernetes.io/projected/0cbe158e-11af-4307-a147-7bc46b788f50-kube-api-access-kmq6v\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:40.410462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.410375    4214 generic.go:356] "Generic (PLEG): container finished" podID="0cbe158e-11af-4307-a147-7bc46b788f50" containerID="2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526" exitCode=137
Oct 30 03:13:40.410462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.410427    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:40.410751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.410473    4214 scope.go:122] "RemoveContainer" containerID="2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526"
Oct 30 03:13:40.412530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.412502    4214 generic.go:356] "Generic (PLEG): container finished" podID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerID="71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0" exitCode=0
Oct 30 03:13:40.412686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.412577    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerDied","Data":"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"}
Oct 30 03:13:40.413048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.413031    4214 scope.go:122] "RemoveContainer" containerID="71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"
Oct 30 03:13:40.414945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.414917    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:40.415035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.414998    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:40.415085 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.415037    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:40.422248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.422219    4214 scope.go:122] "RemoveContainer" containerID="2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526"
Oct 30 03:13:40.422519 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:40.422499    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526\": container with ID starting with 2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526 not found: ID does not exist" containerID="2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526"
Oct 30 03:13:40.422614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.422527    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526"} err="failed to get container status \"2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526\": rpc error: code = NotFound desc = could not find container \"2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526\": container with ID starting with 2c75dad847e3a34e014f837fc8bea44d12a27796241ed472a03decf23a360526 not found: ID does not exist"
Oct 30 03:13:40.469578 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:40.469538    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8809-a1fcaf\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:40.547451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.545404    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:13:40.549392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.548371    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" containerName="test-curl-8809-a1fcaf"
Oct 30 03:13:40.549392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.548399    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" containerName="test-curl-8809-a1fcaf"
Oct 30 03:13:40.549392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.548651    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" containerName="test-curl-8809-a1fcaf"
Oct 30 03:13:40.556272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.552980    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.581119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.571329    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"openshift-service-ca.crt\""
Oct 30 03:13:40.581119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.571558    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"auth-test-model-ac7936-kserve-self-signed-certs\""
Oct 30 03:13:40.581119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.572184    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"default-dockercfg-q26wl\""
Oct 30 03:13:40.581119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.572351    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ac7936\"/\"kube-root-ca.crt\""
Oct 30 03:13:40.581119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.578835    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:13:40.612504 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:40.612459    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8809-a1fcaf\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:40.682617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.682512    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.682796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.682697    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.682796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.682785    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d5h\" (UniqueName: \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.683261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.682859    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.683261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.682900    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.683261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.683081    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.788734    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.788959    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.789074    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.789186    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.789350    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d5h\" (UniqueName: \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.789436    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.790356    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.791027    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.792263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.791669    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.844253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.834895    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d5h\" (UniqueName: \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.844253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.841080    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.844622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.844551    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\") pod \"auth-test-model-ac7936-kserve-6fc7b98f-fcpw5\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") " pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:40.887854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:40.887552    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:41.088104 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:41.080695    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8809-a1fcaf\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" pod="opendatahub/test-curl-8809-a1fcaf"
Oct 30 03:13:41.088104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.087537    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbe158e-11af-4307-a147-7bc46b788f50" path="/var/lib/kubelet/pods/0cbe158e-11af-4307-a147-7bc46b788f50/volumes"
Oct 30 03:13:41.329073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.329043    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:13:41.330469 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:13:41.330438    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fe71af4_84cc_4e82_a458_9eb83195d4d8.slice/crio-ce7fae57866f033dc02dba5adf981f80f7bbe93f69d9f1bec38872304b308f9f WatchSource:0}: Error finding container ce7fae57866f033dc02dba5adf981f80f7bbe93f69d9f1bec38872304b308f9f: Status 404 returned error can't find the container with id ce7fae57866f033dc02dba5adf981f80f7bbe93f69d9f1bec38872304b308f9f
Oct 30 03:13:41.425167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.425133    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerStarted","Data":"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"}
Oct 30 03:13:41.427092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.427065    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" event={"ID":"9fe71af4-84cc-4e82-a458-9eb83195d4d8","Type":"ContainerStarted","Data":"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"}
Oct 30 03:13:41.427220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.427100    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" event={"ID":"9fe71af4-84cc-4e82-a458-9eb83195d4d8","Type":"ContainerStarted","Data":"ce7fae57866f033dc02dba5adf981f80f7bbe93f69d9f1bec38872304b308f9f"}
Oct 30 03:13:41.517761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.517707    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podStartSLOduration=1.517692699 podStartE2EDuration="1.517692699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:41.513039749 +0000 UTC m=+1960.587210542" watchObservedRunningTime="2026-10-30 03:13:41.517692699 +0000 UTC m=+1960.591863492"
Oct 30 03:13:41.821080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.821046    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:13:41.822282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.822219    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:41.822422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.822306    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:41.892253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:41.889333    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:42.821375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:42.821343    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:42.821768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:42.821392    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:43.821912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:43.821845    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:43.821912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:43.821890    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:44.209511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.207209    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:13:44.314526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314498    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.314734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314551    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.314734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314612    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.314734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314629    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.314925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314742    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.314925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314904    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314940    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314951    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314963    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.314964    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.315043    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.315104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.315160    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\") pod \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\" (UID: \"6e4b7242-82ab-4a88-b113-cbc054c7f4cd\") "
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.315532    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.316111    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.316138    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.316152    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.316166    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.316195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.316184    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.317366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.317329    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.317475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.317417    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:13:44.317592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.317573    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:44.317656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.317638    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq" pod "6e4b7242-82ab-4a88-b113-cbc054c7f4cd" (UID: "6e4b7242-82ab-4a88-b113-cbc054c7f4cd"). InnerVolumeSpecName "kube-api-access-qpswq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:44.417868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.417829    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpswq\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-kube-api-access-qpswq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.417868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.417857    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.417868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.417868    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.417868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.417876    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4b7242-82ab-4a88-b113-cbc054c7f4cd-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.503746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.503712    4214 generic.go:356] "Generic (PLEG): container finished" podID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerID="1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a" exitCode=0
Oct 30 03:13:44.503952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.503767    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"
Oct 30 03:13:44.503952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.503792    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerDied","Data":"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"}
Oct 30 03:13:44.503952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.503847    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn" event={"ID":"6e4b7242-82ab-4a88-b113-cbc054c7f4cd","Type":"ContainerDied","Data":"063bd6a69f8a05b32b162f4e11877d538cce863322d1b8b6b8987fb5ea5e1ab1"}
Oct 30 03:13:44.503952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.503869    4214 scope.go:122] "RemoveContainer" containerID="1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"
Oct 30 03:13:44.516933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.516911    4214 scope.go:122] "RemoveContainer" containerID="22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"
Oct 30 03:13:44.528224 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.528205    4214 scope.go:122] "RemoveContainer" containerID="1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"
Oct 30 03:13:44.528595 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:44.528572    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a\": container with ID starting with 1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a not found: ID does not exist" containerID="1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"
Oct 30 03:13:44.528678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.528607    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a"} err="failed to get container status \"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a\": rpc error: code = NotFound desc = could not find container \"1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a\": container with ID starting with 1f5efc61a35b93f2f91a6ff7ed5bdf3eba209bfff03316c770c919ad96ca1a5a not found: ID does not exist"
Oct 30 03:13:44.528678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.528631    4214 scope.go:122] "RemoveContainer" containerID="22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"
Oct 30 03:13:44.528918 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:44.528896    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171\": container with ID starting with 22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171 not found: ID does not exist" containerID="22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"
Oct 30 03:13:44.528963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.528924    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171"} err="failed to get container status \"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171\": rpc error: code = NotFound desc = could not find container \"22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171\": container with ID starting with 22d0d509dfaafbe80db5aac1b1df7a78bf8ca97c9ceb949669be58a5a91ad171 not found: ID does not exist"
Oct 30 03:13:44.579905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.579827    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:13:44.590287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.590257    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-t6zfn"]
Oct 30 03:13:44.821289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.821217    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:44.821471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:44.821328    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:45.086507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:45.086474    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" path="/var/lib/kubelet/pods/6e4b7242-82ab-4a88-b113-cbc054c7f4cd/volumes"
Oct 30 03:13:45.192429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:45.192391    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:13:45.831896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:45.831855    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:45.832083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:45.831929    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:46.822181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:46.822140    4214 patch_prober.go:29] interesting pod/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 03:13:46.822719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:46.822205    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 03:13:47.921749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:47.921715    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:13:47.925182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:47.925147    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:13:50.888992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:50.888953    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:50.902185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:50.902160    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:51.413643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:51.413605    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:13:51.608610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:51.608564    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" podUID="daabb08d-b6e7-4fce-aff8-6572a6412da4" containerName="maas-api" containerID="cri-o://95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43" gracePeriod=30
Oct 30 03:13:51.620710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:51.620685    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:13:52.391826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.391791    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:13:52.396323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.396293    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\") pod \"daabb08d-b6e7-4fce-aff8-6572a6412da4\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") "
Oct 30 03:13:52.396480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.396363    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\") pod \"daabb08d-b6e7-4fce-aff8-6572a6412da4\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") "
Oct 30 03:13:52.396480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.396457    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\" (UniqueName: \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\") pod \"daabb08d-b6e7-4fce-aff8-6572a6412da4\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") "
Oct 30 03:13:52.396687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.396662    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\") pod \"daabb08d-b6e7-4fce-aff8-6572a6412da4\" (UID: \"daabb08d-b6e7-4fce-aff8-6572a6412da4\") "
Oct 30 03:13:52.397137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.397113    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca" pod "daabb08d-b6e7-4fce-aff8-6572a6412da4" (UID: "daabb08d-b6e7-4fce-aff8-6572a6412da4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:52.397669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.397650    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daabb08d-b6e7-4fce-aff8-6572a6412da4-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:52.399473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.399448    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs" pod "daabb08d-b6e7-4fce-aff8-6572a6412da4" (UID: "daabb08d-b6e7-4fce-aff8-6572a6412da4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:52.400657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.400630    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls" pod "daabb08d-b6e7-4fce-aff8-6572a6412da4" (UID: "daabb08d-b6e7-4fce-aff8-6572a6412da4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:52.401535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.401508    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg" pod "daabb08d-b6e7-4fce-aff8-6572a6412da4" (UID: "daabb08d-b6e7-4fce-aff8-6572a6412da4"). InnerVolumeSpecName "kube-api-access-4qzjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:52.499786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.499702    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:52.499786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.499742    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daabb08d-b6e7-4fce-aff8-6572a6412da4-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:52.499786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.499757    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qzjg\" (UniqueName: \"kubernetes.io/projected/daabb08d-b6e7-4fce-aff8-6572a6412da4-kube-api-access-4qzjg\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:13:52.696328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.677452    4214 generic.go:356] "Generic (PLEG): container finished" podID="daabb08d-b6e7-4fce-aff8-6572a6412da4" containerID="95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43" exitCode=0
Oct 30 03:13:52.696328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.678604    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"
Oct 30 03:13:52.696724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.693068    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" event={"ID":"daabb08d-b6e7-4fce-aff8-6572a6412da4","Type":"ContainerDied","Data":"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"}
Oct 30 03:13:52.696724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.696697    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt" event={"ID":"daabb08d-b6e7-4fce-aff8-6572a6412da4","Type":"ContainerDied","Data":"5b7fd3248adc7679d52485304efab4a44a7aab37f22f8b17fc359799c415f510"}
Oct 30 03:13:52.697474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.697446    4214 scope.go:122] "RemoveContainer" containerID="95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"
Oct 30 03:13:52.718833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.718809    4214 scope.go:122] "RemoveContainer" containerID="95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"
Oct 30 03:13:52.719202 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:13:52.719179    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43\": container with ID starting with 95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43 not found: ID does not exist" containerID="95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"
Oct 30 03:13:52.719334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.719244    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43"} err="failed to get container status \"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43\": rpc error: code = NotFound desc = could not find container \"95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43\": container with ID starting with 95cd84c99dc3dbb2daf3685d309ddeac67e1af0a8348ceea80d34583fd06bd43 not found: ID does not exist"
Oct 30 03:13:52.760845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.760749    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:13:52.768684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:52.768650    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-734d43-6c6774fc6d-xlmjt"]
Oct 30 03:13:53.076996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:53.076958    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daabb08d-b6e7-4fce-aff8-6572a6412da4" path="/var/lib/kubelet/pods/daabb08d-b6e7-4fce-aff8-6572a6412da4/volumes"
Oct 30 03:13:57.772101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.772063    4214 generic.go:356] "Generic (PLEG): container finished" podID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerID="42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768" exitCode=0
Oct 30 03:13:57.772931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.772902    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerDied","Data":"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"}
Oct 30 03:13:57.773627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.773602    4214 scope.go:122] "RemoveContainer" containerID="42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"
Oct 30 03:13:57.777207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.777155    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:57.777438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.777414    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:57.777584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:57.777566    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:13:58.784695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:58.784659    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerStarted","Data":"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"}
Oct 30 03:13:59.196252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.175331    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:13:59.407250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.402303    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:13:59.407250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.402939    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" containerID="cri-o://13f400683f5471704a6ec21b5e18958b190e84ff02874d07a8540db3a0958d9e" gracePeriod=30
Oct 30 03:13:59.792254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.781942    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:13:59.792254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.787381    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:13:59.792254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:13:59.787461    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 03:14:00.482815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.482773    4214 patch_prober.go:29] interesting pod/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:14:00.483040 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.482845    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:14:00.681656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.681625    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:14:00.682746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682718    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.682746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682742    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682774    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682780    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682802    4214 cpu_manager.go:414] "containerMap: removing container" podUID="daabb08d-b6e7-4fce-aff8-6572a6412da4" containerName="maas-api"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682808    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daabb08d-b6e7-4fce-aff8-6572a6412da4" containerName="maas-api"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682931    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.682946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682947    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="daabb08d-b6e7-4fce-aff8-6572a6412da4" containerName="maas-api"
Oct 30 03:14:00.683219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.682990    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4b7242-82ab-4a88-b113-cbc054c7f4cd" containerName="istio-proxy"
Oct 30 03:14:00.687965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.687943    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.717898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.717863    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b0b40d-openshift-default-dockercfg-z6nn7\""
Oct 30 03:14:00.720015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.719989    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:14:00.762637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.762535    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:14:00.762637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.762604    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 03:14:00.787121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787081    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787155    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787198    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787266    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787327    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787360    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787390    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554h4\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787452    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.787673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.787649    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.890555    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.890652    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.890922    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891368    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891468    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891528    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891557    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554h4\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891645    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891717    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.891979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.891939    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.908645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.892218    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.908645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.892734    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.909474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.909441    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.934269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.926677    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.949427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.949344    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.949929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.949869    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.950553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.950506    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:00.953626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:00.953600    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554h4\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\") pod \"e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") " pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:01.039909 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.039577    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:01.590997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.590957    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:14:01.591350 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:01.591325    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7e7371_b50b_40a7_be15_1d852d292aa4.slice/crio-bc18e2967786fd2150941248f65289b0cd8cd233ad4d4376ed85f9b89ae1a767 WatchSource:0}: Error finding container bc18e2967786fd2150941248f65289b0cd8cd233ad4d4376ed85f9b89ae1a767: Status 404 returned error can't find the container with id bc18e2967786fd2150941248f65289b0cd8cd233ad4d4376ed85f9b89ae1a767
Oct 30 03:14:01.598515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.596513    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:01.598515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.596670    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:01.598515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.596759    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:01.768288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.767850    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:14:01.768288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.767908    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 03:14:01.873342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.867686    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerStarted","Data":"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"}
Oct 30 03:14:01.873342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.867801    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerStarted","Data":"bc18e2967786fd2150941248f65289b0cd8cd233ad4d4376ed85f9b89ae1a767"}
Oct 30 03:14:01.916615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:01.916561    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podStartSLOduration=1.916547849 podStartE2EDuration="1.916547849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:01.911429621 +0000 UTC m=+1980.985600414" watchObservedRunningTime="2026-10-30 03:14:01.916547849 +0000 UTC m=+1980.990718641"
Oct 30 03:14:02.099252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.090890    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:02.099252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.092396    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:14:02.099252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.092447    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:14:02.761251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.760660    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:14:02.761251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.760724    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 03:14:02.854254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.854040    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:14:02.888257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.887732    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:02.889414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.887631    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:14:02.904001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.903971    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2305f4-serving-cert\""
Oct 30 03:14:02.987018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.986961    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:02.987969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.987931    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bps\" (UniqueName: \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:02.988279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.988262    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:02.988432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:02.988418    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.100309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.099577    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.100309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.099904    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bps\" (UniqueName: \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.100309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.100071    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.100309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.100105    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.131251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.110014    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.131251 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:14:03.110815    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-serving-cert: secret "maas-api-e2e-inf-b-2305f4-serving-cert" not found
Oct 30 03:14:03.131251 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:14:03.110873    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls podName:b0fd346f-62cd-4bba-8f36-52b078f8cd19 nodeName:}" failed. No retries permitted until 2026-10-30 03:14:03.61085553 +0000 UTC m=+1982.685026301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls") pod "maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" (UID: "b0fd346f-62cd-4bba-8f36-52b078f8cd19") : secret "maas-api-e2e-inf-b-2305f4-serving-cert" not found
Oct 30 03:14:03.131251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.114793    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.142895    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bps\" (UniqueName: \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.184255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.183970    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:03.618774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.618736    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.621395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.621367    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") pod \"maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.761786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.761621    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 03:14:03.761786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.761680    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 03:14:03.818083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.818048    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:03.917421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.917392    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:03.918289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:03.918269    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:14:04.181107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.179358    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:14:04.892279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.888121    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:14:04.898494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.898470    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:14:04.930010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.929985    4214 generic.go:356] "Generic (PLEG): container finished" podID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerID="13f400683f5471704a6ec21b5e18958b190e84ff02874d07a8540db3a0958d9e" exitCode=0
Oct 30 03:14:04.930215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.930193    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerDied","Data":"13f400683f5471704a6ec21b5e18958b190e84ff02874d07a8540db3a0958d9e"}
Oct 30 03:14:04.930294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.930272    4214 scope.go:122] "RemoveContainer" containerID="700456c4b704c8e1157a78fe402c309806a9feefd8f01e7e72e0f0e24dfea7ee"
Oct 30 03:14:04.932917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:04.932891    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" event={"ID":"b0fd346f-62cd-4bba-8f36-52b078f8cd19","Type":"ContainerStarted","Data":"f2500cb19d75b9bce7a67e1209fd30c1b61e815c9bf76b7ca526b9aec7642769"}
Oct 30 03:14:05.101933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.101907    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:14:05.233824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.233778    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.233862    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234015    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234048    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234158    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234195    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234281    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234324    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234356    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.234713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.234400    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\") pod \"759dd1fd-8bd1-4282-8847-b948d57c4305\" (UID: \"759dd1fd-8bd1-4282-8847-b948d57c4305\") "
Oct 30 03:14:05.239133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.235219    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.239133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.235599    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:05.239133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.235668    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:05.239133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.235819    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:05.239133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.236103    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:05.239564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.239147    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "kube-api-access-gkppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:05.240865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.240842    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:05.253465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.253400    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:14:05.254281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.254249    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo" pod "759dd1fd-8bd1-4282-8847-b948d57c4305" (UID: "759dd1fd-8bd1-4282-8847-b948d57c4305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:14:05.337008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.336971    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337008    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337025    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkppk\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-kube-api-access-gkppk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337039    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759dd1fd-8bd1-4282-8847-b948d57c4305-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337053    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337066    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759dd1fd-8bd1-4282-8847-b948d57c4305-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337078    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.337195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.337089    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759dd1fd-8bd1-4282-8847-b948d57c4305-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:14:05.945639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.945599    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" event={"ID":"b0fd346f-62cd-4bba-8f36-52b078f8cd19","Type":"ContainerStarted","Data":"8bda2b1f7e8667ec017c7bd7d35e99d9c9651f7c7adf71b1bc2df2c397af9dd1"}
Oct 30 03:14:05.945871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.945709    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:05.948116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.948092    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"
Oct 30 03:14:05.948303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.948121    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b" event={"ID":"759dd1fd-8bd1-4282-8847-b948d57c4305","Type":"ContainerDied","Data":"faa73bb306f2728741e19cd61bd901cdb30f921c46e1a42dcff00c452577feb5"}
Oct 30 03:14:05.948303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.948182    4214 scope.go:122] "RemoveContainer" containerID="13f400683f5471704a6ec21b5e18958b190e84ff02874d07a8540db3a0958d9e"
Oct 30 03:14:05.992449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:05.992395    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" podStartSLOduration=3.450150298 podStartE2EDuration="3.992381778s" totalImagesPullingTime="542.23148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:05.987811256 +0000 UTC m=+1985.061982049" watchObservedRunningTime="2026-10-30 03:14:05.992381778 +0000 UTC m=+1985.066552571"
Oct 30 03:14:06.029223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.029180    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:14:06.033809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.033782    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-734d43-openshift-default-65fd65866f-lmq6b"]
Oct 30 03:14:06.319171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.319141    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:14:06.320252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320216    4214 cpu_manager.go:414] "containerMap: removing container" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.320325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320254    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.320325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320282    4214 cpu_manager.go:414] "containerMap: removing container" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.320325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320287    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.320441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320435    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.320486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.320448    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" containerName="istio-proxy"
Oct 30 03:14:06.326342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.326326    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.341977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.341949    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b0b40d\""
Oct 30 03:14:06.342209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.342103    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b0b40d-dockercfg-ns2j4\""
Oct 30 03:14:06.348950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.348900    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:14:06.504252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.501241    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkmn\" (UniqueName: \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.504252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.501323    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.648289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.630368    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkmn\" (UniqueName: \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.648986 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.648957    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.650183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.650155    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.736256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.732776    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:14:06.748905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.740061    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkmn\" (UniqueName: \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\") pod \"payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.762400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.761865    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:14:06.762400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.761986    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.851992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.851950    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.852216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.852197    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwts\" (UniqueName: \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.945914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.945822    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:06.954593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.954557    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.954780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.954759    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwts\" (UniqueName: \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.955502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.955473    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:06.975709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:06.975681    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwts\" (UniqueName: \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\") pod \"payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:07.076751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.076715    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759dd1fd-8bd1-4282-8847-b948d57c4305" path="/var/lib/kubelet/pods/759dd1fd-8bd1-4282-8847-b948d57c4305/volumes"
Oct 30 03:14:07.085571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.085539    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:07.285143 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:07.285104    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436526a6_cd16_415c_a6b8_168ed98f8d1e.slice/crio-fe341d8affcf7934a24280664b46d14d7d0c560e30b36b68ff44357c76b639cf WatchSource:0}: Error finding container fe341d8affcf7934a24280664b46d14d7d0c560e30b36b68ff44357c76b639cf: Status 404 returned error can't find the container with id fe341d8affcf7934a24280664b46d14d7d0c560e30b36b68ff44357c76b639cf
Oct 30 03:14:07.288288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.287924    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:14:07.493595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.493566    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:14:07.494421 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:07.494399    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e8ccd_e20e_4d9b_9ebb_4614586b0562.slice/crio-deb2df0ba994130c245a8f018e94b3c2ba9817e01aa52245e988a9e59fb294f3 WatchSource:0}: Error finding container deb2df0ba994130c245a8f018e94b3c2ba9817e01aa52245e988a9e59fb294f3: Status 404 returned error can't find the container with id deb2df0ba994130c245a8f018e94b3c2ba9817e01aa52245e988a9e59fb294f3
Oct 30 03:14:07.968244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.968192    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" event={"ID":"436526a6-cd16-415c-a6b8-168ed98f8d1e","Type":"ContainerStarted","Data":"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"}
Oct 30 03:14:07.968436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.968297    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:07.968436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.968322    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" event={"ID":"436526a6-cd16-415c-a6b8-168ed98f8d1e","Type":"ContainerStarted","Data":"fe341d8affcf7934a24280664b46d14d7d0c560e30b36b68ff44357c76b639cf"}
Oct 30 03:14:07.970129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.970102    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" event={"ID":"b60e8ccd-e20e-4d9b-9ebb-4614586b0562","Type":"ContainerStarted","Data":"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"}
Oct 30 03:14:07.970327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.970148    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" event={"ID":"b60e8ccd-e20e-4d9b-9ebb-4614586b0562","Type":"ContainerStarted","Data":"deb2df0ba994130c245a8f018e94b3c2ba9817e01aa52245e988a9e59fb294f3"}
Oct 30 03:14:07.970327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:07.970254    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:08.009054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:08.009007    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" podStartSLOduration=2.008994499 podStartE2EDuration="2.008994499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:08.0077014 +0000 UTC m=+1987.081872193" watchObservedRunningTime="2026-10-30 03:14:08.008994499 +0000 UTC m=+1987.083165291"
Oct 30 03:14:08.055396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:08.055310    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" podStartSLOduration=2.055291104 podStartE2EDuration="2.055291104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:08.051696007 +0000 UTC m=+1987.125866801" watchObservedRunningTime="2026-10-30 03:14:08.055291104 +0000 UTC m=+1987.129461898"
Oct 30 03:14:09.166142 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:14:09.166097    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:14:12.003564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:11.996425    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:14:14.679571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.679527    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:14:14.685557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.685530    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.703792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.703755    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:14:14.708966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.708910    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7f767\"/\"openshift-service-ca.crt\""
Oct 30 03:14:14.708966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.708919    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7f767\"/\"kube-root-ca.crt\""
Oct 30 03:14:14.716266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.713919    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7f767\"/\"test-model-e7f767-kserve-self-signed-certs\""
Oct 30 03:14:14.716266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.713935    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7f767\"/\"default-dockercfg-29sd8\""
Oct 30 03:14:14.733749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.733687    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.733938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.733762    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnxh\" (UniqueName: \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.734009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.733944    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.734296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.734272    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.734467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.734445    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.734584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.734496    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.835820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.835759    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.836345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.835890    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.836345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.835944    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnxh\" (UniqueName: \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.836345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.836077    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.836345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.836304    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.836345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.836336    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.837837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.837808    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.838118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.838098    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.838743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.838404    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.845271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.841149    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.845271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.841626    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:14.859269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:14.858648    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnxh\" (UniqueName: \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\") pod \"test-model-e7f767-kserve-b87db6dc9-m6knv\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") " pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:15.007298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:15.005539    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:15.523216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:15.523173    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:14:16.123665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:16.123626    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" event={"ID":"f62b93e7-a3b6-462c-876b-470fe471d080","Type":"ContainerStarted","Data":"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"}
Oct 30 03:14:16.124096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:16.123674    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" event={"ID":"f62b93e7-a3b6-462c-876b-470fe471d080","Type":"ContainerStarted","Data":"8c9d8abfbec93d3c1ff29521f8ac01c49ee04ce71b5c6365b4a53a5a3dd52b9b"}
Oct 30 03:14:16.169056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:16.168996    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podStartSLOduration=2.168976006 podStartE2EDuration="2.168976006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:16.162665992 +0000 UTC m=+1995.236836786" watchObservedRunningTime="2026-10-30 03:14:16.168976006 +0000 UTC m=+1995.243146800"
Oct 30 03:14:17.011254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:17.006316    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:18.985072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:18.985037    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:14:18.985529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:18.985120    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:14:22.786253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.777293    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:14:22.935296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.935201    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.010253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.983738    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-100ffd\"/\"default-dockercfg-dgtwd\""
Oct 30 03:14:23.010253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.984327    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-100ffd\"/\"test-model-100ffd-kserve-self-signed-certs\""
Oct 30 03:14:23.010253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.984387    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-100ffd\"/\"kube-root-ca.crt\""
Oct 30 03:14:23.010253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:22.984475    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-100ffd\"/\"openshift-service-ca.crt\""
Oct 30 03:14:23.155255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.134449    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.155609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.155578    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.155838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.155821    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.155959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.155945    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.156135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.156118    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xg5\" (UniqueName: \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.194367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.194326    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.300729    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.300816    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.300926    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.300966    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.301041    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xg5\" (UniqueName: \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.301080    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.307633    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.308127    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.320135    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.335252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.321399    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:14:23.340983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.340955    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.382255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.354023    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xg5\" (UniqueName: \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.387927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.387873    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\") pod \"test-model-100ffd-kserve-7bfbb5bdf-jgj7r\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") " pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.412191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.411896    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:23.654253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.653316    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.230:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:23.729361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.722372    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:23.931280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:23.909221    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main" probeResult="failure" output="Get \"https://10.133.0.226:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:24.474179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:24.474148    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:14:24.474861 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:24.474832    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84dd794b_07e2_443e_880a_8e1d0360334a.slice/crio-eb92cfedc5b5e75667a571e4d01f6c329a2ebc9d785061a8ce0c171164f64d66 WatchSource:0}: Error finding container eb92cfedc5b5e75667a571e4d01f6c329a2ebc9d785061a8ce0c171164f64d66: Status 404 returned error can't find the container with id eb92cfedc5b5e75667a571e4d01f6c329a2ebc9d785061a8ce0c171164f64d66
Oct 30 03:14:25.006656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.006618    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:25.019655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.019623    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:25.374798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.374760    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" event={"ID":"84dd794b-07e2-443e-880a-8e1d0360334a","Type":"ContainerStarted","Data":"fb261a3baa9c379da96485db69b09ceaeb6b395df05ff9c75e34c95cec808c2d"}
Oct 30 03:14:25.374993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.374809    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" event={"ID":"84dd794b-07e2-443e-880a-8e1d0360334a","Type":"ContainerStarted","Data":"eb92cfedc5b5e75667a571e4d01f6c329a2ebc9d785061a8ce0c171164f64d66"}
Oct 30 03:14:25.387334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.387302    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:14:25.418364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.418301    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podStartSLOduration=3.418284414 podStartE2EDuration="3.418284414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:25.413843976 +0000 UTC m=+2004.488014771" watchObservedRunningTime="2026-10-30 03:14:25.418284414 +0000 UTC m=+2004.492455210"
Oct 30 03:14:25.419850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:25.419827    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:32.260335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.260297    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:14:32.265627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.265604    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.287037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.287004    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:14:32.294255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.293265    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4f415d-openshift-default-dockercfg-qkscz\""
Oct 30 03:14:32.375111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375073    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375176    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375208    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hgz\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375315    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375361    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375464    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375570    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375623    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.375685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.375653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.477876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.477838    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478039    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478089    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478120    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hgz\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478185    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478255    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478336    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478442    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478473    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478485    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478564    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.478864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.478831    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.479059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.479036    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.479137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.479117    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.480811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.480789    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.481182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.481159    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.498455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.498429    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.499006 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.498976    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hgz\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\") pod \"e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") " pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.582415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.582373    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:32.936891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.936857    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:14:32.971521 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:32.971471    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c40987_aae2_40e2_8ccc_d0552ad44e85.slice/crio-94b8a5c2901c1d7c7aa0fb14556514ea59f9d37f5490283698768d255dd84d69 WatchSource:0}: Error finding container 94b8a5c2901c1d7c7aa0fb14556514ea59f9d37f5490283698768d255dd84d69: Status 404 returned error can't find the container with id 94b8a5c2901c1d7c7aa0fb14556514ea59f9d37f5490283698768d255dd84d69
Oct 30 03:14:32.999921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.999870    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:33.000091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:32.999971    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:33.000091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.000014    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:14:33.420133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.420097    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:33.433944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.433919    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:33.485141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.485095    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" event={"ID":"b3c40987-aae2-40e2-8ccc-d0552ad44e85","Type":"ContainerStarted","Data":"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"}
Oct 30 03:14:33.485350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.485154    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" event={"ID":"b3c40987-aae2-40e2-8ccc-d0552ad44e85","Type":"ContainerStarted","Data":"94b8a5c2901c1d7c7aa0fb14556514ea59f9d37f5490283698768d255dd84d69"}
Oct 30 03:14:33.497748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.497722    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:14:33.528928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.528874    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podStartSLOduration=1.5288557520000001 podStartE2EDuration="1.528855752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:33.526468217 +0000 UTC m=+2012.600639009" watchObservedRunningTime="2026-10-30 03:14:33.528855752 +0000 UTC m=+2012.603026551"
Oct 30 03:14:33.582667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:33.582637    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:34.591283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:34.589977    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:34.591283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:34.590058    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:35.582952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:35.582912    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:35.583179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:35.582968    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:36.583822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:36.583780    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:36.584372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:36.583839    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:37.583170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:37.583132    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:37.583488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:37.583455    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:38.583346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:38.583311    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:38.583809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:38.583783    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:39.309908 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:14:39.309859    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:14:39.586607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:39.586516    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 03:14:39.586607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:39.586594    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 03:14:40.670469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:40.670419    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:40.671004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:40.670976    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:40.672952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:40.672929    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:14:44.643636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.643554    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:14:44.651097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.651072    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.669656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.669623    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:14:44.673883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.673386    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.673883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.673527    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.674316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.674289    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.674443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.674340    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.674443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.674364    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b0b40d-serving-cert\""
Oct 30 03:14:44.776559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.776523    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.776775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.776568    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.776775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.776676    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.776775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.776756    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.777684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.777652    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.779341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.779319    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.779666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.779641    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.797844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.797824    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\") pod \"maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:44.985301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:44.980784    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:45.357640 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:14:45.357610    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d09a3c_2ec5_422d_93ad_cf6c3aeee3ec.slice/crio-9facad2b393595261e46878ac59e9f4133452a09d50952f5dd191edaaa3dfa27 WatchSource:0}: Error finding container 9facad2b393595261e46878ac59e9f4133452a09d50952f5dd191edaaa3dfa27: Status 404 returned error can't find the container with id 9facad2b393595261e46878ac59e9f4133452a09d50952f5dd191edaaa3dfa27
Oct 30 03:14:45.357759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:45.357639    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:14:45.658191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:45.658101    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" event={"ID":"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec","Type":"ContainerStarted","Data":"9facad2b393595261e46878ac59e9f4133452a09d50952f5dd191edaaa3dfa27"}
Oct 30 03:14:46.669300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:46.669257    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" event={"ID":"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec","Type":"ContainerStarted","Data":"a9a57e02dfe65636d8aac81f22b4ac5cd288958f50c8096a18496b61447b7311"}
Oct 30 03:14:46.669798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:46.669328    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:14:46.758256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:46.734193    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" podStartSLOduration=2.294410037 podStartE2EDuration="2.734174959s" totalImagesPullingTime="439.764922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:46.733301613 +0000 UTC m=+2025.807472409" watchObservedRunningTime="2026-10-30 03:14:46.734174959 +0000 UTC m=+2025.808345753"
Oct 30 03:14:52.689027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:14:52.688979    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:15:00.260253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.236307    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:00.260253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.242984    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:00.260253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.243163    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.260253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.249872    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.260253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.256835    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:00.274642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.271945    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:00.279100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.278755    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:00.290647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.287286    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.290647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.288997    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:00.295539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.295447    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:15:00.300260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.299406    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:00.306875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.304441    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.314246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.308685    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.314246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.308802    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:00.332749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.325808    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.384721    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9hw\" (UniqueName: \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.384794    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pl6\" (UniqueName: \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.384835    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjc4\" (UniqueName: \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.384932    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftbj\" (UniqueName: \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.384965    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.385005    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwvm\" (UniqueName: \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.385047    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.385131    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.385193    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.389333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.385422    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.434317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.420182    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.553901    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9hw\" (UniqueName: \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.553967    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pl6\" (UniqueName: \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554000    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjc4\" (UniqueName: \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554083    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftbj\" (UniqueName: \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554119    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554159    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwvm\" (UniqueName: \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554201    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554290    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554347    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.554522    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.555426    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.558375    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.559091    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.559798    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.576100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.559961    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.583190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.583091    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjc4\" (UniqueName: \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\") pod \"maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.583666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.583642    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9hw\" (UniqueName: \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\") pod \"maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.584072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.584046    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pl6\" (UniqueName: \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.584616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.584594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwvm\" (UniqueName: \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\") pod \"maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.585426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.585375    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftbj\" (UniqueName: \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.598721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.598536    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:00.627153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.627123    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:00.723161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.723121    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:00.743873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.743850    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:00.910599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.906212    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:00.972328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:00.972296    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:01.193800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:01.193765    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:01.250945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:01.250915    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:01.342174 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:15:01.342139    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b31064_04b9_4ecd_a7da_e785332c93d9.slice/crio-57ad014bb6975c236b974615a8ce14afe22fa22d608f5068151af6087fa808b7 WatchSource:0}: Error finding container 57ad014bb6975c236b974615a8ce14afe22fa22d608f5068151af6087fa808b7: Status 404 returned error can't find the container with id 57ad014bb6975c236b974615a8ce14afe22fa22d608f5068151af6087fa808b7
Oct 30 03:15:01.350143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:01.350121    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:01.438972 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:15:01.438937    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bf0b28_ef66_4a56_97ed_68b4254092b0.slice/crio-ae7c6efe8a77dc0d7474a81fbe310ff9e1bccf09cc5e6388f7c77675d1c80356 WatchSource:0}: Error finding container ae7c6efe8a77dc0d7474a81fbe310ff9e1bccf09cc5e6388f7c77675d1c80356: Status 404 returned error can't find the container with id ae7c6efe8a77dc0d7474a81fbe310ff9e1bccf09cc5e6388f7c77675d1c80356
Oct 30 03:15:01.458212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:01.458179    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:01.538378 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:15:01.538326    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388f6b73_1004_46b4_80a2_badc595341c8.slice/crio-10594ebce3a8bcb2f928cc023ef616ac11f63ab947547d80b312b3f45ac3b9b6 WatchSource:0}: Error finding container 10594ebce3a8bcb2f928cc023ef616ac11f63ab947547d80b312b3f45ac3b9b6: Status 404 returned error can't find the container with id 10594ebce3a8bcb2f928cc023ef616ac11f63ab947547d80b312b3f45ac3b9b6
Oct 30 03:15:02.038106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.038016    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerStarted","Data":"e114c06bead826325663da6fc210d6cbafcbb187232ba25d4342796c40aac745"}
Oct 30 03:15:02.038106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.038084    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerStarted","Data":"57ad014bb6975c236b974615a8ce14afe22fa22d608f5068151af6087fa808b7"}
Oct 30 03:15:02.040873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.040812    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerStarted","Data":"ae7c6efe8a77dc0d7474a81fbe310ff9e1bccf09cc5e6388f7c77675d1c80356"}
Oct 30 03:15:02.043011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.042975    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerStarted","Data":"451597eac0aee2d65e84e46dee4f2c3f0d028d1b11bf36fb5fb7d0b832ecace1"}
Oct 30 03:15:02.043142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.043039    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerStarted","Data":"12eb39a74f0de714f419e5574f9d5384b87091f78c945b235937a2ef692abe22"}
Oct 30 03:15:02.048459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.047722    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerStarted","Data":"10594ebce3a8bcb2f928cc023ef616ac11f63ab947547d80b312b3f45ac3b9b6"}
Oct 30 03:15:02.049668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.049643    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerStarted","Data":"93c8df52fbfb37f52779438912129776b59295e7592ead3a99d628ed5dd4bf9f"}
Oct 30 03:15:02.049771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.049683    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerStarted","Data":"d3877df5161d7e62f642c32b956e60360059b7cdd4c683047c94736151379e58"}
Oct 30 03:15:02.148655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.148058    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" podStartSLOduration=2.148029972 podStartE2EDuration="2.148029972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.140629231 +0000 UTC m=+2041.214800026" watchObservedRunningTime="2026-10-30 03:15:02.148029972 +0000 UTC m=+2041.222200769"
Oct 30 03:15:02.154085 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.154026    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" podStartSLOduration=2.154012927 podStartE2EDuration="2.154012927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.099954648 +0000 UTC m=+2041.174125442" watchObservedRunningTime="2026-10-30 03:15:02.154012927 +0000 UTC m=+2041.228183766"
Oct 30 03:15:02.212403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:02.212346    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" podStartSLOduration=2.212328499 podStartE2EDuration="2.212328499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.189848361 +0000 UTC m=+2041.264019153" watchObservedRunningTime="2026-10-30 03:15:02.212328499 +0000 UTC m=+2041.286499293"
Oct 30 03:15:03.064554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.064512    4214 generic.go:356] "Generic (PLEG): container finished" podID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerID="451597eac0aee2d65e84e46dee4f2c3f0d028d1b11bf36fb5fb7d0b832ecace1" exitCode=60
Oct 30 03:15:03.065098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.065078    4214 scope.go:122] "RemoveContainer" containerID="451597eac0aee2d65e84e46dee4f2c3f0d028d1b11bf36fb5fb7d0b832ecace1"
Oct 30 03:15:03.073065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.073036    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerDied","Data":"451597eac0aee2d65e84e46dee4f2c3f0d028d1b11bf36fb5fb7d0b832ecace1"}
Oct 30 03:15:03.087575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.085433    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerStarted","Data":"df5c1d2e01cde7af3b4cc8edb65336b2b09f20d6a0dd49d89f1171dc1cacf0da"}
Oct 30 03:15:03.089673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.089545    4214 generic.go:356] "Generic (PLEG): container finished" podID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerID="93c8df52fbfb37f52779438912129776b59295e7592ead3a99d628ed5dd4bf9f" exitCode=60
Oct 30 03:15:03.090138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.089730    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerDied","Data":"93c8df52fbfb37f52779438912129776b59295e7592ead3a99d628ed5dd4bf9f"}
Oct 30 03:15:03.090138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.089895    4214 scope.go:122] "RemoveContainer" containerID="93c8df52fbfb37f52779438912129776b59295e7592ead3a99d628ed5dd4bf9f"
Oct 30 03:15:03.092335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.092310    4214 generic.go:356] "Generic (PLEG): container finished" podID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerID="e114c06bead826325663da6fc210d6cbafcbb187232ba25d4342796c40aac745" exitCode=60
Oct 30 03:15:03.092482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.092457    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerDied","Data":"e114c06bead826325663da6fc210d6cbafcbb187232ba25d4342796c40aac745"}
Oct 30 03:15:03.092654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.092629    4214 scope.go:122] "RemoveContainer" containerID="e114c06bead826325663da6fc210d6cbafcbb187232ba25d4342796c40aac745"
Oct 30 03:15:03.095820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.095782    4214 generic.go:356] "Generic (PLEG): container finished" podID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerID="e198d4d107336e7491ba4ec6c9a1e45fa0e2050ec6918317367e095d96ead1dd" exitCode=60
Oct 30 03:15:03.095894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.095852    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerDied","Data":"e198d4d107336e7491ba4ec6c9a1e45fa0e2050ec6918317367e095d96ead1dd"}
Oct 30 03:15:03.096222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.096204    4214 scope.go:122] "RemoveContainer" containerID="e198d4d107336e7491ba4ec6c9a1e45fa0e2050ec6918317367e095d96ead1dd"
Oct 30 03:15:03.198307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:03.198181    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" podStartSLOduration=3.198163451 podStartE2EDuration="3.198163451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.194915301 +0000 UTC m=+2042.269086096" watchObservedRunningTime="2026-10-30 03:15:03.198163451 +0000 UTC m=+2042.272334242"
Oct 30 03:15:04.124747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.124698    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerStarted","Data":"c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"}
Oct 30 03:15:04.127642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.127611    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerStarted","Data":"f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"}
Oct 30 03:15:04.129787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.129757    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerStarted","Data":"63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"}
Oct 30 03:15:04.132422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.132364    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerStarted","Data":"969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"}
Oct 30 03:15:04.134143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.134119    4214 generic.go:356] "Generic (PLEG): container finished" podID="388f6b73-1004-46b4-80a2-badc595341c8" containerID="df5c1d2e01cde7af3b4cc8edb65336b2b09f20d6a0dd49d89f1171dc1cacf0da" exitCode=60
Oct 30 03:15:04.134251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.134168    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerDied","Data":"df5c1d2e01cde7af3b4cc8edb65336b2b09f20d6a0dd49d89f1171dc1cacf0da"}
Oct 30 03:15:04.134467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.134442    4214 scope.go:122] "RemoveContainer" containerID="df5c1d2e01cde7af3b4cc8edb65336b2b09f20d6a0dd49d89f1171dc1cacf0da"
Oct 30 03:15:04.291415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.291377    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:15:04.291632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.291434    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:04.333986 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:04.333910    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" podStartSLOduration=4.3338569719999995 podStartE2EDuration="4.333856972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:04.330533844 +0000 UTC m=+2043.404704628" watchObservedRunningTime="2026-10-30 03:15:04.333856972 +0000 UTC m=+2043.408027765"
Oct 30 03:15:05.151738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.151699    4214 generic.go:356] "Generic (PLEG): container finished" podID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerID="f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1" exitCode=60
Oct 30 03:15:05.152275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.151900    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerDied","Data":"f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"}
Oct 30 03:15:05.152275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.151955    4214 scope.go:122] "RemoveContainer" containerID="e114c06bead826325663da6fc210d6cbafcbb187232ba25d4342796c40aac745"
Oct 30 03:15:05.152275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.152155    4214 scope.go:122] "RemoveContainer" containerID="f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"
Oct 30 03:15:05.152617 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:05.152584    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl_odh-ai-gateway-infra(81b31064-04b9-4ecd-a7da-e785332c93d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9"
Oct 30 03:15:05.156211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.156163    4214 generic.go:356] "Generic (PLEG): container finished" podID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerID="63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab" exitCode=60
Oct 30 03:15:05.156457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.156432    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerDied","Data":"63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"}
Oct 30 03:15:05.156647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.156625    4214 scope.go:122] "RemoveContainer" containerID="63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"
Oct 30 03:15:05.157007 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:05.156984    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9_odh-ai-gateway-infra(83bf0b28-ef66-4a56-97ed-68b4254092b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0"
Oct 30 03:15:05.159190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.159165    4214 generic.go:356] "Generic (PLEG): container finished" podID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerID="969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3" exitCode=60
Oct 30 03:15:05.159478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.159458    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerDied","Data":"969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"}
Oct 30 03:15:05.161136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.160829    4214 scope.go:122] "RemoveContainer" containerID="969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"
Oct 30 03:15:05.161314 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:05.161272    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9_odh-ai-gateway-infra(b90ed7f9-ebaf-4841-944d-c6f677a1bf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a"
Oct 30 03:15:05.163455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.163424    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerStarted","Data":"b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"}
Oct 30 03:15:05.167529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.166816    4214 generic.go:356] "Generic (PLEG): container finished" podID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerID="c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4" exitCode=60
Oct 30 03:15:05.167529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.166871    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerDied","Data":"c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"}
Oct 30 03:15:05.167529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.167248    4214 scope.go:122] "RemoveContainer" containerID="c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"
Oct 30 03:15:05.167783 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:05.167559    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5_odh-ai-gateway-infra(5350992a-dde7-47b0-9f06-d2c9acd77a76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76"
Oct 30 03:15:05.172684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.172659    4214 scope.go:122] "RemoveContainer" containerID="e198d4d107336e7491ba4ec6c9a1e45fa0e2050ec6918317367e095d96ead1dd"
Oct 30 03:15:05.192678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.192658    4214 scope.go:122] "RemoveContainer" containerID="451597eac0aee2d65e84e46dee4f2c3f0d028d1b11bf36fb5fb7d0b832ecace1"
Oct 30 03:15:05.207859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:05.207839    4214 scope.go:122] "RemoveContainer" containerID="93c8df52fbfb37f52779438912129776b59295e7592ead3a99d628ed5dd4bf9f"
Oct 30 03:15:06.182917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.182633    4214 generic.go:356] "Generic (PLEG): container finished" podID="388f6b73-1004-46b4-80a2-badc595341c8" containerID="b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2" exitCode=60
Oct 30 03:15:06.182917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.182783    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerDied","Data":"b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"}
Oct 30 03:15:06.182917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.182842    4214 scope.go:122] "RemoveContainer" containerID="df5c1d2e01cde7af3b4cc8edb65336b2b09f20d6a0dd49d89f1171dc1cacf0da"
Oct 30 03:15:06.185378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.183121    4214 scope.go:122] "RemoveContainer" containerID="b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"
Oct 30 03:15:06.185378 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:06.183650    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct_odh-ai-gateway-infra(388f6b73-1004-46b4-80a2-badc595341c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" podUID="388f6b73-1004-46b4-80a2-badc595341c8"
Oct 30 03:15:06.188716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.188692    4214 scope.go:122] "RemoveContainer" containerID="c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"
Oct 30 03:15:06.189094 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:06.189070    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5_odh-ai-gateway-infra(5350992a-dde7-47b0-9f06-d2c9acd77a76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76"
Oct 30 03:15:06.204988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.204969    4214 scope.go:122] "RemoveContainer" containerID="f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"
Oct 30 03:15:06.205333 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:06.205303    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl_odh-ai-gateway-infra(81b31064-04b9-4ecd-a7da-e785332c93d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9"
Oct 30 03:15:06.207113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.207093    4214 scope.go:122] "RemoveContainer" containerID="63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"
Oct 30 03:15:06.207448 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:06.207423    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9_odh-ai-gateway-infra(83bf0b28-ef66-4a56-97ed-68b4254092b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0"
Oct 30 03:15:06.208927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:06.208910    4214 scope.go:122] "RemoveContainer" containerID="969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"
Oct 30 03:15:06.209217 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:06.209199    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9_odh-ai-gateway-infra(b90ed7f9-ebaf-4841-944d-c6f677a1bf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a"
Oct 30 03:15:07.219643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.219610    4214 scope.go:122] "RemoveContainer" containerID="b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"
Oct 30 03:15:07.220189 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:07.220156    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct_odh-ai-gateway-infra(388f6b73-1004-46b4-80a2-badc595341c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" podUID="388f6b73-1004-46b4-80a2-badc595341c8"
Oct 30 03:15:07.221150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.221124    4214 generic.go:356] "Generic (PLEG): container finished" podID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerID="c770c796ace441054a6dbd444f2baee2ef64a7e359a1a157d0aedec409b8211c" exitCode=0
Oct 30 03:15:07.221265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.221156    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerDied","Data":"c770c796ace441054a6dbd444f2baee2ef64a7e359a1a157d0aedec409b8211c"}
Oct 30 03:15:07.221563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.221548    4214 scope.go:122] "RemoveContainer" containerID="c770c796ace441054a6dbd444f2baee2ef64a7e359a1a157d0aedec409b8211c"
Oct 30 03:15:07.223637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.223607    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:07.223726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.223677    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:07.223726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:07.223709    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:08.231683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:08.231646    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerStarted","Data":"2b581d3a6e70e7bb65bc667f1d152f559a1aae2e6a0532dc0ec310033e7369bd"}
Oct 30 03:15:08.811881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:08.811779    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:15:08.812102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:08.811936    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 03:15:08.812102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:08.811994    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 03:15:09.195705 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:09.195618    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:15:09.814637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:09.814603    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 03:15:09.815072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:09.814684    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 03:15:10.816699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:10.812071    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 03:15:10.816699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:10.812131    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 03:15:11.811297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:11.811261    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 03:15:11.811471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:11.811313    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 03:15:12.811958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:12.811921    4214 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 03:15:12.812467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:12.811976    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 03:15:13.816661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:13.816631    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:15:13.817144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:13.817090    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:15:13.817894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:13.817867    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:15:15.059162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:15.059127    4214 scope.go:122] "RemoveContainer" containerID="63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"
Oct 30 03:15:16.059623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:16.059598    4214 scope.go:122] "RemoveContainer" containerID="c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"
Oct 30 03:15:16.436282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:16.432218    4214 generic.go:356] "Generic (PLEG): container finished" podID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerID="5a8a88dd9c0215483a8e9948daee9ea5fc942edb6a3d638eee2551d942b2bfa7" exitCode=60
Oct 30 03:15:16.436282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:16.432298    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerDied","Data":"5a8a88dd9c0215483a8e9948daee9ea5fc942edb6a3d638eee2551d942b2bfa7"}
Oct 30 03:15:16.436282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:16.432376    4214 scope.go:122] "RemoveContainer" containerID="63ec18de285a1e36aa3061da3eac4934e7be80ce9e88969720ff8836e75fe8ab"
Oct 30 03:15:16.436282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:16.433088    4214 scope.go:122] "RemoveContainer" containerID="5a8a88dd9c0215483a8e9948daee9ea5fc942edb6a3d638eee2551d942b2bfa7"
Oct 30 03:15:16.436282 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:16.433560    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9_odh-ai-gateway-infra(83bf0b28-ef66-4a56-97ed-68b4254092b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0"
Oct 30 03:15:17.059505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.059473    4214 scope.go:122] "RemoveContainer" containerID="f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"
Oct 30 03:15:17.448748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.448721    4214 generic.go:356] "Generic (PLEG): container finished" podID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerID="74f678529be38707949d334866880f62d31b19a9670d517cdc245e7ebe20e2d3" exitCode=60
Oct 30 03:15:17.449101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.448790    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerDied","Data":"74f678529be38707949d334866880f62d31b19a9670d517cdc245e7ebe20e2d3"}
Oct 30 03:15:17.449101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.448853    4214 scope.go:122] "RemoveContainer" containerID="c47dc6fc14a97a886e4713b3b501f20964dc3d0519f40843ba7c02049355c5e4"
Oct 30 03:15:17.449368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.449351    4214 scope.go:122] "RemoveContainer" containerID="74f678529be38707949d334866880f62d31b19a9670d517cdc245e7ebe20e2d3"
Oct 30 03:15:17.449789 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:17.449765    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5_odh-ai-gateway-infra(5350992a-dde7-47b0-9f06-d2c9acd77a76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76"
Oct 30 03:15:17.489761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.489733    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:17.695021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.694986    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:17.789465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.789426    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\" (UniqueName: \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\") pod \"83bf0b28-ef66-4a56-97ed-68b4254092b0\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") "
Oct 30 03:15:17.789724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.789698    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\") pod \"83bf0b28-ef66-4a56-97ed-68b4254092b0\" (UID: \"83bf0b28-ef66-4a56-97ed-68b4254092b0\") "
Oct 30 03:15:17.790249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.790207    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca" pod "83bf0b28-ef66-4a56-97ed-68b4254092b0" (UID: "83bf0b28-ef66-4a56-97ed-68b4254092b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:17.790462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.790443    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf0b28-ef66-4a56-97ed-68b4254092b0-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.798291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.798265    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm" pod "83bf0b28-ef66-4a56-97ed-68b4254092b0" (UID: "83bf0b28-ef66-4a56-97ed-68b4254092b0"). InnerVolumeSpecName "kube-api-access-ftwvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:17.891576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:17.891544    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwvm\" (UniqueName: \"kubernetes.io/projected/83bf0b28-ef66-4a56-97ed-68b4254092b0-kube-api-access-ftwvm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.059512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.059490    4214 scope.go:122] "RemoveContainer" containerID="969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"
Oct 30 03:15:18.461684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.461653    4214 generic.go:356] "Generic (PLEG): container finished" podID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerID="f945e87689298578d98608b3a0feb2b993746c3eac8186695459cd7d5ba876c5" exitCode=60
Oct 30 03:15:18.462059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.461736    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerDied","Data":"f945e87689298578d98608b3a0feb2b993746c3eac8186695459cd7d5ba876c5"}
Oct 30 03:15:18.462059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.461820    4214 scope.go:122] "RemoveContainer" containerID="f3c6d189bb3ff3055cb04511938bae28eeca99c88cdd7098175a87699ac28fc1"
Oct 30 03:15:18.462209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.462196    4214 scope.go:122] "RemoveContainer" containerID="f945e87689298578d98608b3a0feb2b993746c3eac8186695459cd7d5ba876c5"
Oct 30 03:15:18.462641 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:18.462617    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl_odh-ai-gateway-infra(81b31064-04b9-4ecd-a7da-e785332c93d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9"
Oct 30 03:15:18.463587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.463565    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"
Oct 30 03:15:18.463709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.463585    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9" event={"ID":"83bf0b28-ef66-4a56-97ed-68b4254092b0","Type":"ContainerDied","Data":"ae7c6efe8a77dc0d7474a81fbe310ff9e1bccf09cc5e6388f7c77675d1c80356"}
Oct 30 03:15:18.476543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.476521    4214 scope.go:122] "RemoveContainer" containerID="5a8a88dd9c0215483a8e9948daee9ea5fc942edb6a3d638eee2551d942b2bfa7"
Oct 30 03:15:18.510089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.510064    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:18.549638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.549604    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:18.554757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.554730    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7f767-29888835-mgfm9"]
Oct 30 03:15:18.783760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.783733    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:18.910446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.910403    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\" (UniqueName: \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\") pod \"5350992a-dde7-47b0-9f06-d2c9acd77a76\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") "
Oct 30 03:15:18.910640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.910487    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\") pod \"5350992a-dde7-47b0-9f06-d2c9acd77a76\" (UID: \"5350992a-dde7-47b0-9f06-d2c9acd77a76\") "
Oct 30 03:15:18.911681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.911652    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca" pod "5350992a-dde7-47b0-9f06-d2c9acd77a76" (UID: "5350992a-dde7-47b0-9f06-d2c9acd77a76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:18.914439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:18.914414    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj" pod "5350992a-dde7-47b0-9f06-d2c9acd77a76" (UID: "5350992a-dde7-47b0-9f06-d2c9acd77a76"). InnerVolumeSpecName "kube-api-access-jftbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:19.012835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.012753    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jftbj\" (UniqueName: \"kubernetes.io/projected/5350992a-dde7-47b0-9f06-d2c9acd77a76-kube-api-access-jftbj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:19.012835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.012779    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5350992a-dde7-47b0-9f06-d2c9acd77a76-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:19.070071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.070035    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" path="/var/lib/kubelet/pods/83bf0b28-ef66-4a56-97ed-68b4254092b0/volumes"
Oct 30 03:15:19.071396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.071372    4214 scope.go:122] "RemoveContainer" containerID="b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"
Oct 30 03:15:19.482154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.482127    4214 generic.go:356] "Generic (PLEG): container finished" podID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerID="f26d47a64cd31ee41f3de16af35e5d7836711d399487eb16e78bcc1cb6029167" exitCode=60
Oct 30 03:15:19.482631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.482198    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerDied","Data":"f26d47a64cd31ee41f3de16af35e5d7836711d399487eb16e78bcc1cb6029167"}
Oct 30 03:15:19.482631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.482290    4214 scope.go:122] "RemoveContainer" containerID="969149339d3977434bd28095f732f15e7b7dadb155fc4d396282d0a7f1b32ec3"
Oct 30 03:15:19.482751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.482732    4214 scope.go:122] "RemoveContainer" containerID="f26d47a64cd31ee41f3de16af35e5d7836711d399487eb16e78bcc1cb6029167"
Oct 30 03:15:19.483216 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:19.483169    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9_odh-ai-gateway-infra(b90ed7f9-ebaf-4841-944d-c6f677a1bf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a"
Oct 30 03:15:19.484465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.484440    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerStarted","Data":"2f91887e1cb9e72031542c8a34468c7804e4210a6432b25b6fdba667423940d3"}
Oct 30 03:15:19.485917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.485897    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"
Oct 30 03:15:19.486034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.485917    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5" event={"ID":"5350992a-dde7-47b0-9f06-d2c9acd77a76","Type":"ContainerDied","Data":"d3877df5161d7e62f642c32b956e60360059b7cdd4c683047c94736151379e58"}
Oct 30 03:15:19.503019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.503002    4214 scope.go:122] "RemoveContainer" containerID="74f678529be38707949d334866880f62d31b19a9670d517cdc245e7ebe20e2d3"
Oct 30 03:15:19.518951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.518929    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:19.640249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.640140    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:19.660284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.648368    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ac7936-29888835-kqdd5"]
Oct 30 03:15:19.851877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.851793    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:19.960251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.960023    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\") pod \"81b31064-04b9-4ecd-a7da-e785332c93d9\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") "
Oct 30 03:15:19.960424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.960269    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\" (UniqueName: \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\") pod \"81b31064-04b9-4ecd-a7da-e785332c93d9\" (UID: \"81b31064-04b9-4ecd-a7da-e785332c93d9\") "
Oct 30 03:15:19.961529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.961262    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca" pod "81b31064-04b9-4ecd-a7da-e785332c93d9" (UID: "81b31064-04b9-4ecd-a7da-e785332c93d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:19.964510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:19.964480    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw" pod "81b31064-04b9-4ecd-a7da-e785332c93d9" (UID: "81b31064-04b9-4ecd-a7da-e785332c93d9"). InnerVolumeSpecName "kube-api-access-bv9hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.063720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.063687    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9hw\" (UniqueName: \"kubernetes.io/projected/81b31064-04b9-4ecd-a7da-e785332c93d9-kube-api-access-bv9hw\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.063720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.063718    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b31064-04b9-4ecd-a7da-e785332c93d9-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.500148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.500068    4214 generic.go:356] "Generic (PLEG): container finished" podID="388f6b73-1004-46b4-80a2-badc595341c8" containerID="2f91887e1cb9e72031542c8a34468c7804e4210a6432b25b6fdba667423940d3" exitCode=60
Oct 30 03:15:20.500148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.500118    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerDied","Data":"2f91887e1cb9e72031542c8a34468c7804e4210a6432b25b6fdba667423940d3"}
Oct 30 03:15:20.500692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.500188    4214 scope.go:122] "RemoveContainer" containerID="b98612675d9ab0074a677005e14da1d42be16c26d6bb21111c234c8d3f2b9ef2"
Oct 30 03:15:20.500692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.500636    4214 scope.go:122] "RemoveContainer" containerID="2f91887e1cb9e72031542c8a34468c7804e4210a6432b25b6fdba667423940d3"
Oct 30 03:15:20.501088 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:20.501061    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct_odh-ai-gateway-infra(388f6b73-1004-46b4-80a2-badc595341c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" podUID="388f6b73-1004-46b4-80a2-badc595341c8"
Oct 30 03:15:20.504212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.504187    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl" event={"ID":"81b31064-04b9-4ecd-a7da-e785332c93d9","Type":"ContainerDied","Data":"57ad014bb6975c236b974615a8ce14afe22fa22d608f5068151af6087fa808b7"}
Oct 30 03:15:20.504344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.504304    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"
Oct 30 03:15:20.525186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.525164    4214 scope.go:122] "RemoveContainer" containerID="f945e87689298578d98608b3a0feb2b993746c3eac8186695459cd7d5ba876c5"
Oct 30 03:15:20.560571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.560536    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:20.688197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.687626    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:20.695435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.695408    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-100ffd-29888835-5m4sl"]
Oct 30 03:15:20.709998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.709857    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:20.797369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.797341    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:20.980856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.980822    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\") pod \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") "
Oct 30 03:15:20.981101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.980888    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\" (UniqueName: \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\") pod \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\" (UID: \"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a\") "
Oct 30 03:15:20.981284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.981219    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca" pod "b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" (UID: "b90ed7f9-ebaf-4841-944d-c6f677a1bf7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.981789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.981765    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.983292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:20.983269    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4" pod "b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" (UID: "b90ed7f9-ebaf-4841-944d-c6f677a1bf7a"). InnerVolumeSpecName "kube-api-access-xzjc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.072800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.072758    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" path="/var/lib/kubelet/pods/5350992a-dde7-47b0-9f06-d2c9acd77a76/volumes"
Oct 30 03:15:21.073295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.073277    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" path="/var/lib/kubelet/pods/81b31064-04b9-4ecd-a7da-e785332c93d9/volumes"
Oct 30 03:15:21.083156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.083128    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjc4\" (UniqueName: \"kubernetes.io/projected/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a-kube-api-access-xzjc4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.541587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.539977    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9" event={"ID":"b90ed7f9-ebaf-4841-944d-c6f677a1bf7a","Type":"ContainerDied","Data":"12eb39a74f0de714f419e5574f9d5384b87091f78c945b235937a2ef692abe22"}
Oct 30 03:15:21.541587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.540046    4214 scope.go:122] "RemoveContainer" containerID="f26d47a64cd31ee41f3de16af35e5d7836711d399487eb16e78bcc1cb6029167"
Oct 30 03:15:21.541587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.540134    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"
Oct 30 03:15:21.666060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.665967    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:21.679077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.679035    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e11f03-29888835-bnzj9"]
Oct 30 03:15:21.775955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.775928    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:21.896186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.896146    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\") pod \"388f6b73-1004-46b4-80a2-badc595341c8\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") "
Oct 30 03:15:21.896430 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.896304    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\" (UniqueName: \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\") pod \"388f6b73-1004-46b4-80a2-badc595341c8\" (UID: \"388f6b73-1004-46b4-80a2-badc595341c8\") "
Oct 30 03:15:21.896598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.896570    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca" pod "388f6b73-1004-46b4-80a2-badc595341c8" (UID: "388f6b73-1004-46b4-80a2-badc595341c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.897033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.897012    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/388f6b73-1004-46b4-80a2-badc595341c8-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.898557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.898531    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6" pod "388f6b73-1004-46b4-80a2-badc595341c8" (UID: "388f6b73-1004-46b4-80a2-badc595341c8"). InnerVolumeSpecName "kube-api-access-45pl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.998825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:21.998790    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pl6\" (UniqueName: \"kubernetes.io/projected/388f6b73-1004-46b4-80a2-badc595341c8-kube-api-access-45pl6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.567864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:22.567838    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"
Oct 30 03:15:22.568393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:22.567874    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct" event={"ID":"388f6b73-1004-46b4-80a2-badc595341c8","Type":"ContainerDied","Data":"10594ebce3a8bcb2f928cc023ef616ac11f63ab947547d80b312b3f45ac3b9b6"}
Oct 30 03:15:22.568393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:22.567933    4214 scope.go:122] "RemoveContainer" containerID="2f91887e1cb9e72031542c8a34468c7804e4210a6432b25b6fdba667423940d3"
Oct 30 03:15:22.630158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:22.630124    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:22.644759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:22.644733    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d9f3ac-29888835-x64ct"]
Oct 30 03:15:23.070087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:23.070051    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388f6b73-1004-46b4-80a2-badc595341c8" path="/var/lib/kubelet/pods/388f6b73-1004-46b4-80a2-badc595341c8/volumes"
Oct 30 03:15:23.070552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:23.070536    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" path="/var/lib/kubelet/pods/b90ed7f9-ebaf-4841-944d-c6f677a1bf7a/volumes"
Oct 30 03:15:32.259251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.258274    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:15:32.263842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.263810    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.264135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.264116    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.264261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.264249    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.264368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.264356    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.280173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280147    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.280380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280364    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.280500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280488    4214 cpu_manager.go:414] "containerMap: removing container" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.280582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280572    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.280682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280671    4214 cpu_manager.go:414] "containerMap: removing container" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.280759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280749    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.280851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280842    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.280926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280917    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.281005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.280993    4214 cpu_manager.go:414] "containerMap: removing container" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.281252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281220    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.281383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281371    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.281468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281458    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.281591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281582    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.281668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281659    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.281749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281738    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.281834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281824    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.281970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.281956    4214 cpu_manager.go:414] "containerMap: removing container" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282055    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282159    4214 cpu_manager.go:414] "containerMap: removing container" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282254    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282588    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.282719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282707    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.282803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282794    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282876    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.282974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.282965    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.283067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283057    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.283153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283144    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.283252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283223    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:15:32.283357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283347    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.283440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283431    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b31064-04b9-4ecd-a7da-e785332c93d9" containerName="cleanup"
Oct 30 03:15:32.283525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283514    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:15:32.283608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283598    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:15:32.283708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.283698    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90ed7f9-ebaf-4841-944d-c6f677a1bf7a" containerName="cleanup"
Oct 30 03:15:32.304467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.304435    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.349732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.342582    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4cf8f5-openshift-default-dockercfg-zhgfj\""
Oct 30 03:15:32.498855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.456449    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.499162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.499134    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc9r\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.499566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.499533    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.499746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.499724    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.499906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.499889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.500346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.500322    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.500584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.500492    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.500929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.500908    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.501305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.501284    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.549314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.545270    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:15:32.603620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.603444    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.603620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.603492    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.603620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.603545    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.603702    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604039    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604089    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604350    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604393    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604420    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc9r\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604506    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.604769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604750    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.605007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.604988    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.605496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.605222    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.605671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.605615    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.607360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.607338    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.607509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.607472    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.624467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.624449    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.624757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.624730    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc9r\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\") pod \"e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") " pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:32.798891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:32.798855    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:33.190682 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:15:33.190652    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931d8eb0_6f34_4a25_886d_851eff161cd0.slice/crio-0cfc1bacd53c64cc3c1a126e97e050b89a684f81ad8892323e01a3ae26ab91bf WatchSource:0}: Error finding container 0cfc1bacd53c64cc3c1a126e97e050b89a684f81ad8892323e01a3ae26ab91bf: Status 404 returned error can't find the container with id 0cfc1bacd53c64cc3c1a126e97e050b89a684f81ad8892323e01a3ae26ab91bf
Oct 30 03:15:33.190849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.190749    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:15:33.193655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.193626    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:33.193729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.193694    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:33.193729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.193725    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:33.727357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.727317    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerStarted","Data":"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"}
Oct 30 03:15:33.727850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.727373    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerStarted","Data":"0cfc1bacd53c64cc3c1a126e97e050b89a684f81ad8892323e01a3ae26ab91bf"}
Oct 30 03:15:33.800595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.800562    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:33.825366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:33.800865    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podStartSLOduration=1.800845947 podStartE2EDuration="1.800845947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:33.782786965 +0000 UTC m=+2072.856957759" watchObservedRunningTime="2026-10-30 03:15:33.800845947 +0000 UTC m=+2072.875016738"
Oct 30 03:15:34.051307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:34.024154    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:15:34.805303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:34.805273    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:35.820976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:35.820884    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:35.823706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:35.823668    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:15:39.171245 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:39.171192    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:15:48.390251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.387357    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.401210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.401170    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:15:48.412628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.412414    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.674388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.577429    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.674388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.639474    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.230:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.725208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.725161    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:15:48.791401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.791336    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.835437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.835375    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main" probeResult="failure" output="Get \"https://10.133.0.226:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.881549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.881022    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 03:15:48.986304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.891379    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.986304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:48.956219    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main" probeResult="failure" output="Get \"https://10.133.0.226:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.494265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.488708    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.494265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.492368    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.494265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.492672    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 30 03:15:49.494265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.493885    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:15:49.494929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.494333    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:15:49.599265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.598326    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": context deadline exceeded"
Oct 30 03:15:49.599265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.598628    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:15:49.640375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.640321    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.230:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.660273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.642134    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:15:49.811308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.797641    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.811308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.807852    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:15:49.881251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.881153    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.891668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.881646    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:15:49.916384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.916339    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main" probeResult="failure" output="Get \"https://10.133.0.226:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.917532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.917507    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:15:49.999256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:49.973924    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:50.000366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.000339    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:15:50.367253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.352124    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:15:50.492337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.492292    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 03:15:50.493610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.493560    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:50.493821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.493794    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": context deadline exceeded"
Oct 30 03:15:50.494295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.494274    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:15:50.524551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.524528    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:15:50.525198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.525181    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:15:50.608171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.603879    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:50.621311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.615522    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:15:50.636316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.636279    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main" containerID="cri-o://343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96" gracePeriod=44
Oct 30 03:15:50.657859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.657824    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.230:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:50.668256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.665824    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:15:50.676255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.675454    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:15:50.689371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.679874    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:15:50.698476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.696194    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:15:50.698476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:50.696858    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:15:51.290134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.290102    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:15:51.404595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404512    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.404595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404584    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.404827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404739    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.404926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404893    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:51.404926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404913    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.405155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.404982    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:51.405155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405004    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.405155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405074    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\" (UniqueName: \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\") pod \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\" (UID: \"9fe71af4-84cc-4e82-a458-9eb83195d4d8\") "
Oct 30 03:15:51.405394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405164    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:51.405894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405870    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.406009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405901    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.406009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.405915    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.408813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.408789    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:51.408813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.408804    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:51.410014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.409991    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h" pod "9fe71af4-84cc-4e82-a458-9eb83195d4d8" (UID: "9fe71af4-84cc-4e82-a458-9eb83195d4d8"). InnerVolumeSpecName "kube-api-access-n4d5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:51.508212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.508175    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fe71af4-84cc-4e82-a458-9eb83195d4d8-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.508741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.508721    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4d5h\" (UniqueName: \"kubernetes.io/projected/9fe71af4-84cc-4e82-a458-9eb83195d4d8-kube-api-access-n4d5h\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.508835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.508823    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fe71af4-84cc-4e82-a458-9eb83195d4d8-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:15:51.529089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.527321    4214 generic.go:356] "Generic (PLEG): container finished" podID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerID="343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96" exitCode=0
Oct 30 03:15:51.529089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.527380    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"
Oct 30 03:15:51.529089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.527471    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" event={"ID":"9fe71af4-84cc-4e82-a458-9eb83195d4d8","Type":"ContainerDied","Data":"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"}
Oct 30 03:15:51.529089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.527505    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5" event={"ID":"9fe71af4-84cc-4e82-a458-9eb83195d4d8","Type":"ContainerDied","Data":"ce7fae57866f033dc02dba5adf981f80f7bbe93f69d9f1bec38872304b308f9f"}
Oct 30 03:15:51.529089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.527525    4214 scope.go:122] "RemoveContainer" containerID="343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"
Oct 30 03:15:51.589519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.589496    4214 scope.go:122] "RemoveContainer" containerID="343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"
Oct 30 03:15:51.590045 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:15:51.589910    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96\": container with ID starting with 343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96 not found: ID does not exist" containerID="343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"
Oct 30 03:15:51.590045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.589952    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96"} err="failed to get container status \"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96\": rpc error: code = NotFound desc = could not find container \"343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96\": container with ID starting with 343d2eef206a189694064b4f2719ae9436bc872fcd319e4007e14e6a90e6bd96 not found: ID does not exist"
Oct 30 03:15:51.668849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.668762    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:15:51.678656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:51.678619    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ac7936/auth-test-model-ac7936-kserve-6fc7b98f-fcpw5"]
Oct 30 03:15:53.071477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:53.071438    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" path="/var/lib/kubelet/pods/9fe71af4-84cc-4e82-a458-9eb83195d4d8/volumes"
Oct 30 03:15:59.701890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.701852    4214 generic.go:356] "Generic (PLEG): container finished" podID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerID="24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c" exitCode=0
Oct 30 03:15:59.702452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.701919    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerDied","Data":"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"}
Oct 30 03:15:59.702452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.702417    4214 scope.go:122] "RemoveContainer" containerID="24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"
Oct 30 03:15:59.704769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.704736    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:59.704857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.704826    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:15:59.704919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:15:59.704873    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:16:00.714460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:00.714418    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerStarted","Data":"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"}
Oct 30 03:16:01.060262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:01.060136    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:16:01.071255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:01.069347    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:16:01.071255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:01.069423    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:16:02.056254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:02.051047    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:16:02.056254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:02.051117    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:16:03.042254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:03.041082    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:16:03.042254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:03.041150    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:16:03.917342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:03.917307    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:16:04.055254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:04.043583    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:16:04.055254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:04.043647    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:16:05.042796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:05.042761    4214 patch_prober.go:29] interesting pod/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 03:16:05.043452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:05.043405    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 03:16:06.082219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:06.082185    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:16:06.084513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:06.084487    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:16:09.167942 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:16:09.167902    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:16:32.098595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:32.098559    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:16:39.170072 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:16:39.170033    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:16:47.127764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.127707    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main" containerID="cri-o://043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07" gracePeriod=45
Oct 30 03:16:47.442080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.442040    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.489934    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.489993    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\" (UniqueName: \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.490033    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.490139    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.490362    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.490465    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\") pod \"6f2139de-66e7-4906-8dfb-35015be69017\" (UID: \"6f2139de-66e7-4906-8dfb-35015be69017\") "
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.491708    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.491737    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:47.493892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.493707    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:47.494572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.494224    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:47.503818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.503790    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "kube-api-access-srk68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:47.510346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.509905    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs" pod "6f2139de-66e7-4906-8dfb-35015be69017" (UID: "6f2139de-66e7-4906-8dfb-35015be69017"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:47.532534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.532504    4214 generic.go:356] "Generic (PLEG): container finished" podID="6f2139de-66e7-4906-8dfb-35015be69017" containerID="043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07" exitCode=0
Oct 30 03:16:47.532698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.532545    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"
Oct 30 03:16:47.532698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.532586    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" event={"ID":"6f2139de-66e7-4906-8dfb-35015be69017","Type":"ContainerDied","Data":"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"}
Oct 30 03:16:47.532698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.532631    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4" event={"ID":"6f2139de-66e7-4906-8dfb-35015be69017","Type":"ContainerDied","Data":"09e0240ece739eb4cddc4ac195a8603125e83ec705d07dfe6c53603dfcfe3df4"}
Oct 30 03:16:47.532698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.532652    4214 scope.go:122] "RemoveContainer" containerID="043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"
Oct 30 03:16:47.575717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.575606    4214 scope.go:122] "RemoveContainer" containerID="043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"
Oct 30 03:16:47.576045 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:16:47.576015    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07\": container with ID starting with 043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07 not found: ID does not exist" containerID="043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"
Oct 30 03:16:47.576142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.576059    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07"} err="failed to get container status \"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07\": rpc error: code = NotFound desc = could not find container \"043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07\": container with ID starting with 043f72c84c13d1f6e6ba959fe779df71ae24cd490cf6f2ff2616a3b16a03fc07 not found: ID does not exist"
Oct 30 03:16:47.593618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593589    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.593618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593618    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srk68\" (UniqueName: \"kubernetes.io/projected/6f2139de-66e7-4906-8dfb-35015be69017-kube-api-access-srk68\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.593793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593629    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.593793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593639    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2139de-66e7-4906-8dfb-35015be69017-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.593793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593648    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.593793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.593655    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2139de-66e7-4906-8dfb-35015be69017-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:16:47.595404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.595390    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:16:47.600356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:47.600330    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c1fce3/test-model-c1fce3-kserve-7c6c7b94bf-lb5z4"]
Oct 30 03:16:49.071636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:16:49.071603    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2139de-66e7-4906-8dfb-35015be69017" path="/var/lib/kubelet/pods/6f2139de-66e7-4906-8dfb-35015be69017/volumes"
Oct 30 03:17:01.851113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.851072    4214 generic.go:356] "Generic (PLEG): container finished" podID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerID="77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f" exitCode=0
Oct 30 03:17:01.851595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.851141    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerDied","Data":"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"}
Oct 30 03:17:01.851695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.851678    4214 scope.go:122] "RemoveContainer" containerID="77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"
Oct 30 03:17:01.853637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.853607    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:01.853764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.853692    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:01.853764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:01.853746    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:17:02.866609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:02.866570    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerStarted","Data":"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"}
Oct 30 03:17:03.799205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:03.799165    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:17:03.800432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:03.800405    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:17:03.800514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:03.800484    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:17:04.822254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:04.819336    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:17:04.822254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:04.819395    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:17:05.799309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:05.799271    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:17:05.799499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:05.799334    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:17:05.817292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:05.817259    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:17:06.799557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:06.799523    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:17:06.799953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:06.799594    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:17:07.799145 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:07.799109    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:17:07.799377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:07.799184    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:17:09.205996 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:17:09.205881    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:17:09.247670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:09.247158    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:17:09.250048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:09.250013    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:17:15.523947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:15.523905    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:17:15.559181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:15.559145    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:17:20.525359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:20.525311    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing" containerID="cri-o://d5e02e05ac6caa2a02c7381df4496fc26db0772131bd903f9919809fcad7137b" gracePeriod=25
Oct 30 03:17:20.559675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:20.559633    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing" containerID="cri-o://dff7cfcb2f1ecbd533e103085b49080f5a530ad9ff055b0775c9b1772d3bc5b9" gracePeriod=25
Oct 30 03:17:23.296540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:23.296494    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:17:24.550993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:24.550947    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:17:26.628705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:26.628674    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:17:26.629151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:26.628904    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" podUID="57cd6b97-58e0-4803-9145-9673538a3546" containerName="maas-api" containerID="cri-o://d3e0fe6864f4409b383f39524001107d6184bc1f69e464b09d017edbf2df3dcc" gracePeriod=30
Oct 30 03:17:27.346625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.346580    4214 generic.go:356] "Generic (PLEG): container finished" podID="57cd6b97-58e0-4803-9145-9673538a3546" containerID="d3e0fe6864f4409b383f39524001107d6184bc1f69e464b09d017edbf2df3dcc" exitCode=0
Oct 30 03:17:27.346910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.346890    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" event={"ID":"57cd6b97-58e0-4803-9145-9673538a3546","Type":"ContainerDied","Data":"d3e0fe6864f4409b383f39524001107d6184bc1f69e464b09d017edbf2df3dcc"}
Oct 30 03:17:27.404371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.404341    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:17:27.569349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.569317    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\") pod \"57cd6b97-58e0-4803-9145-9673538a3546\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") "
Oct 30 03:17:27.569535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.569504    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") pod \"57cd6b97-58e0-4803-9145-9673538a3546\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") "
Oct 30 03:17:27.569612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.569575    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\") pod \"57cd6b97-58e0-4803-9145-9673538a3546\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") "
Oct 30 03:17:27.569828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.569803    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\" (UniqueName: \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\") pod \"57cd6b97-58e0-4803-9145-9673538a3546\" (UID: \"57cd6b97-58e0-4803-9145-9673538a3546\") "
Oct 30 03:17:27.570011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.569982    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca" pod "57cd6b97-58e0-4803-9145-9673538a3546" (UID: "57cd6b97-58e0-4803-9145-9673538a3546"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:27.570871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.570847    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57cd6b97-58e0-4803-9145-9673538a3546-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:27.571870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.571840    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs" pod "57cd6b97-58e0-4803-9145-9673538a3546" (UID: "57cd6b97-58e0-4803-9145-9673538a3546"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:27.572129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.572104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls" pod "57cd6b97-58e0-4803-9145-9673538a3546" (UID: "57cd6b97-58e0-4803-9145-9673538a3546"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:17:27.573449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.573424    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872" pod "57cd6b97-58e0-4803-9145-9673538a3546" (UID: "57cd6b97-58e0-4803-9145-9673538a3546"). InnerVolumeSpecName "kube-api-access-jv872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:27.686011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.685912    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:27.686011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.685954    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57cd6b97-58e0-4803-9145-9673538a3546-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:27.686011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:27.685970    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv872\" (UniqueName: \"kubernetes.io/projected/57cd6b97-58e0-4803-9145-9673538a3546-kube-api-access-jv872\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:28.360678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:28.360646    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"
Oct 30 03:17:28.360678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:28.360668    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc" event={"ID":"57cd6b97-58e0-4803-9145-9673538a3546","Type":"ContainerDied","Data":"dd0d224705d5e98ad978f5f2a6b7f37ccd36fa15d3247bf37faa6f7704674ebe"}
Oct 30 03:17:28.360976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:28.360736    4214 scope.go:122] "RemoveContainer" containerID="d3e0fe6864f4409b383f39524001107d6184bc1f69e464b09d017edbf2df3dcc"
Oct 30 03:17:28.421363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:28.421329    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:17:28.433785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:28.433758    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1fce3-5d865dc54b-ttdwc"]
Oct 30 03:17:29.072652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:29.072608    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cd6b97-58e0-4803-9145-9673538a3546" path="/var/lib/kubelet/pods/57cd6b97-58e0-4803-9145-9673538a3546/volumes"
Oct 30 03:17:32.289267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:32.289213    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:17:32.289701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:32.289456    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy" containerID="cri-o://c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187" gracePeriod=30
Oct 30 03:17:33.297704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:33.297659    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:17:33.971075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:33.971034    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:17:33.971328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:33.971285    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy" containerID="cri-o://d7db3b1e5220b5d1d5570e4ec9e24ce65cf92eed50f6b41a1012ce231fc35f96" gracePeriod=30
Oct 30 03:17:34.595417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:34.595367    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:17:34.866055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:34.865956    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:17:34.866335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:34.866282    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy" containerID="cri-o://ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484" gracePeriod=30
Oct 30 03:17:37.632249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.630883    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:17:37.644888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.644847    4214 generic.go:356] "Generic (PLEG): container finished" podID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerID="c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187" exitCode=0
Oct 30 03:17:37.645076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.645058    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" event={"ID":"3dbe6f81-793d-4a2f-be93-856cf3405fbf","Type":"ContainerDied","Data":"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"}
Oct 30 03:17:37.645128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.645113    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng" event={"ID":"3dbe6f81-793d-4a2f-be93-856cf3405fbf","Type":"ContainerDied","Data":"d0bdb34fe2af534b68b36077769000a874fff167888c3b104ded99221bc34c5b"}
Oct 30 03:17:37.645165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.645144    4214 scope.go:122] "RemoveContainer" containerID="c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"
Oct 30 03:17:37.645335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.645322    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"
Oct 30 03:17:37.663517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.663494    4214 scope.go:122] "RemoveContainer" containerID="c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"
Oct 30 03:17:37.663885 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:17:37.663850    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187\": container with ID starting with c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187 not found: ID does not exist" containerID="c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"
Oct 30 03:17:37.664019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.663893    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187"} err="failed to get container status \"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187\": rpc error: code = NotFound desc = could not find container \"c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187\": container with ID starting with c7bb431a792765728ca5ee803b5b7325c51135589e0b7a20eca6e621f098f187 not found: ID does not exist"
Oct 30 03:17:37.702828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.702792    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.702911    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.702941    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703059    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703117    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703173    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703250    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703355    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703393    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\") pod \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\" (UID: \"3dbe6f81-793d-4a2f-be93-856cf3405fbf\") "
Oct 30 03:17:37.703539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703429    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:37.703539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.703459    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:37.704443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.704415    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.704578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.704447    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.706408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.705545    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:37.706408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.706324    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:37.715108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.711301    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:37.715212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.715123    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:37.716571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.716540    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:37.716780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.716753    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:37.716920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.716839    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97" pod "3dbe6f81-793d-4a2f-be93-856cf3405fbf" (UID: "3dbe6f81-793d-4a2f-be93-856cf3405fbf"). InnerVolumeSpecName "kube-api-access-s6s97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:37.811763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811689    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.811763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811727    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.811763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811743    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.811763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811756    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.812089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811769    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbe6f81-793d-4a2f-be93-856cf3405fbf-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.812089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811790    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6s97\" (UniqueName: \"kubernetes.io/projected/3dbe6f81-793d-4a2f-be93-856cf3405fbf-kube-api-access-s6s97\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.812089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.811804    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbe6f81-793d-4a2f-be93-856cf3405fbf-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:37.999819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:37.999789    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:17:38.012243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:38.012192    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zprng"]
Oct 30 03:17:39.074065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.074025    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" path="/var/lib/kubelet/pods/3dbe6f81-793d-4a2f-be93-856cf3405fbf/volumes"
Oct 30 03:17:39.172258 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:17:39.170049    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:17:39.686277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.686173    4214 generic.go:356] "Generic (PLEG): container finished" podID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerID="d7db3b1e5220b5d1d5570e4ec9e24ce65cf92eed50f6b41a1012ce231fc35f96" exitCode=0
Oct 30 03:17:39.686277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.686214    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" event={"ID":"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd","Type":"ContainerDied","Data":"d7db3b1e5220b5d1d5570e4ec9e24ce65cf92eed50f6b41a1012ce231fc35f96"}
Oct 30 03:17:39.713189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.713166    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:17:39.851806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851536    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.851806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851705    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.851806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851796    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851874    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851928    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.851976    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.852321    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.852435    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.852821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.852577    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\") pod \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\" (UID: \"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd\") "
Oct 30 03:17:39.853292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.853191    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:39.853528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.853506    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:39.853625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.853599    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:39.853741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.853703    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:39.853812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.853780    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:39.854765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.854738    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.854885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.854765    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.854885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.854781    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.854885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.854794    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.854885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.854807    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.858486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.858457    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "kube-api-access-ltph7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:39.858830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.858803    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:39.863903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.863357    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:39.863903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.863808    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo" pod "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" (UID: "dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:39.956808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.956772    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.956808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.956810    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltph7\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-kube-api-access-ltph7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.957096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.956825    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:39.957096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:39.956840    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.229725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.229699    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:17:40.368693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.368655    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.368895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.368719    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.368895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.368794    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.368895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.368830    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.368956    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.369089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369032    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369063    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.369247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369102    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369178    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369308    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369368    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\") pod \"c9c9d962-30dd-4ec5-9c07-0470736111d2\" (UID: \"c9c9d962-30dd-4ec5-9c07-0470736111d2\") "
Oct 30 03:17:40.369496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369468    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.369614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369544    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.369896    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.370198    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c9d962-30dd-4ec5-9c07-0470736111d2-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.370219    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.370280    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.370296    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.370334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.370309    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.371662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.371636    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:40.371778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.371696    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "kube-api-access-fml9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.371980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.371954    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.372065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.372029    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token" pod "c9c9d962-30dd-4ec5-9c07-0470736111d2" (UID: "c9c9d962-30dd-4ec5-9c07-0470736111d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.471731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.471654    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.471731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.471685    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.471731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.471698    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fml9m\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-kube-api-access-fml9m\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.471731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.471709    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c9d962-30dd-4ec5-9c07-0470736111d2-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.708342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.708300    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb" event={"ID":"dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd","Type":"ContainerDied","Data":"3047b996af70157651e62692f56ebe5c0b1b00e230e06dd478b1aad74aecb5de"}
Oct 30 03:17:40.708488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.708375    4214 scope.go:122] "RemoveContainer" containerID="d7db3b1e5220b5d1d5570e4ec9e24ce65cf92eed50f6b41a1012ce231fc35f96"
Oct 30 03:17:40.708488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.708314    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"
Oct 30 03:17:40.717305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.717276    4214 generic.go:356] "Generic (PLEG): container finished" podID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerID="dff7cfcb2f1ecbd533e103085b49080f5a530ad9ff055b0775c9b1772d3bc5b9" exitCode=0
Oct 30 03:17:40.717474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.717457    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" event={"ID":"ff44dcf2-81d8-4044-8ea8-a664383f888b","Type":"ContainerDied","Data":"dff7cfcb2f1ecbd533e103085b49080f5a530ad9ff055b0775c9b1772d3bc5b9"}
Oct 30 03:17:40.727116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.727085    4214 generic.go:356] "Generic (PLEG): container finished" podID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerID="ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484" exitCode=0
Oct 30 03:17:40.727290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.727143    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"
Oct 30 03:17:40.727367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.727306    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerDied","Data":"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"}
Oct 30 03:17:40.727425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.727365    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm" event={"ID":"c9c9d962-30dd-4ec5-9c07-0470736111d2","Type":"ContainerDied","Data":"e921ac8deea726465fdbf1ea05634344994b7d7c1735600142ba52b345abcd74"}
Oct 30 03:17:40.729920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.729893    4214 generic.go:356] "Generic (PLEG): container finished" podID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerID="d5e02e05ac6caa2a02c7381df4496fc26db0772131bd903f9919809fcad7137b" exitCode=0
Oct 30 03:17:40.730110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.730092    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" event={"ID":"b9290e3e-cc98-4f16-b348-109b43f92e30","Type":"ContainerDied","Data":"d5e02e05ac6caa2a02c7381df4496fc26db0772131bd903f9919809fcad7137b"}
Oct 30 03:17:40.734600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.734578    4214 scope.go:122] "RemoveContainer" containerID="ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"
Oct 30 03:17:40.757482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.757440    4214 scope.go:122] "RemoveContainer" containerID="480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"
Oct 30 03:17:40.767331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.766320    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:17:40.778730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.778707    4214 scope.go:122] "RemoveContainer" containerID="ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"
Oct 30 03:17:40.779073 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:17:40.779035    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484\": container with ID starting with ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484 not found: ID does not exist" containerID="ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"
Oct 30 03:17:40.779167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.779074    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484"} err="failed to get container status \"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484\": rpc error: code = NotFound desc = could not find container \"ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484\": container with ID starting with ea2cb8d643aa63849c9f87afbbc969ca9b88ce49a8e69ab4b86f3b257f849484 not found: ID does not exist"
Oct 30 03:17:40.779167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.779101    4214 scope.go:122] "RemoveContainer" containerID="480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"
Oct 30 03:17:40.779527 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:17:40.779459    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91\": container with ID starting with 480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91 not found: ID does not exist" containerID="480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"
Oct 30 03:17:40.779527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.779498    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91"} err="failed to get container status \"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91\": rpc error: code = NotFound desc = could not find container \"480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91\": container with ID starting with 480d45051bf18145b873ad18fb14553a18800b3c67e4c906430f4df214a5ed91 not found: ID does not exist"
Oct 30 03:17:40.784258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.784202    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\") pod \"b9290e3e-cc98-4f16-b348-109b43f92e30\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") "
Oct 30 03:17:40.784367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.784272    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\" (UniqueName: \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\") pod \"b9290e3e-cc98-4f16-b348-109b43f92e30\" (UID: \"b9290e3e-cc98-4f16-b348-109b43f92e30\") "
Oct 30 03:17:40.786092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.785752    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume" pod "b9290e3e-cc98-4f16-b348-109b43f92e30" (UID: "b9290e3e-cc98-4f16-b348-109b43f92e30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:40.788181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.788142    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt" pod "b9290e3e-cc98-4f16-b348-109b43f92e30" (UID: "b9290e3e-cc98-4f16-b348-109b43f92e30"). InnerVolumeSpecName "kube-api-access-hvwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.821027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.820753    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:17:40.828917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.828884    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-g6dmb"]
Oct 30 03:17:40.863770 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.863736    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:17:40.874039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.874002    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-kzgpm"]
Oct 30 03:17:40.886657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.886620    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9290e3e-cc98-4f16-b348-109b43f92e30-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.886657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.886659    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvwkt\" (UniqueName: \"kubernetes.io/projected/b9290e3e-cc98-4f16-b348-109b43f92e30-kube-api-access-hvwkt\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.909678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.909653    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:17:40.987666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.987584    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\") pod \"ff44dcf2-81d8-4044-8ea8-a664383f888b\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") "
Oct 30 03:17:40.987883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.987860    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\" (UniqueName: \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\") pod \"ff44dcf2-81d8-4044-8ea8-a664383f888b\" (UID: \"ff44dcf2-81d8-4044-8ea8-a664383f888b\") "
Oct 30 03:17:40.987970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.987949    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume" pod "ff44dcf2-81d8-4044-8ea8-a664383f888b" (UID: "ff44dcf2-81d8-4044-8ea8-a664383f888b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:40.988697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.988669    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff44dcf2-81d8-4044-8ea8-a664383f888b-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.990510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:40.990481    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv" pod "ff44dcf2-81d8-4044-8ea8-a664383f888b" (UID: "ff44dcf2-81d8-4044-8ea8-a664383f888b"). InnerVolumeSpecName "kube-api-access-n97sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:41.072726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.072693    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" path="/var/lib/kubelet/pods/c9c9d962-30dd-4ec5-9c07-0470736111d2/volumes"
Oct 30 03:17:41.073405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.073385    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" path="/var/lib/kubelet/pods/dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd/volumes"
Oct 30 03:17:41.089813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.089792    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n97sv\" (UniqueName: \"kubernetes.io/projected/ff44dcf2-81d8-4044-8ea8-a664383f888b-kube-api-access-n97sv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:17:41.743223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.743188    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"
Oct 30 03:17:41.743223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.743247    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7" event={"ID":"b9290e3e-cc98-4f16-b348-109b43f92e30","Type":"ContainerDied","Data":"40c062a3640de49af079ea3e3d67fb30f476efeb4cc01ca0fd0c06bc7f58f00a"}
Oct 30 03:17:41.743830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.743319    4214 scope.go:122] "RemoveContainer" containerID="d5e02e05ac6caa2a02c7381df4496fc26db0772131bd903f9919809fcad7137b"
Oct 30 03:17:41.746800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.746766    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"
Oct 30 03:17:41.746927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.746767    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn" event={"ID":"ff44dcf2-81d8-4044-8ea8-a664383f888b","Type":"ContainerDied","Data":"50910e25791f311ebfe6814fa2788ba73fe2ce92460b532edd821e5e1e58f9c6"}
Oct 30 03:17:41.759437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.759417    4214 scope.go:122] "RemoveContainer" containerID="dff7cfcb2f1ecbd533e103085b49080f5a530ad9ff055b0775c9b1772d3bc5b9"
Oct 30 03:17:41.800104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.800070    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:17:41.812864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.812787    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1fce3-7cc6d59d47-c67hn"]
Oct 30 03:17:41.849746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.849711    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:17:41.854690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:41.854661    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1fce3-bcbf85d6b-w92w7"]
Oct 30 03:17:43.070552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.070515    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" path="/var/lib/kubelet/pods/b9290e3e-cc98-4f16-b348-109b43f92e30/volumes"
Oct 30 03:17:43.071062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.071045    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" path="/var/lib/kubelet/pods/ff44dcf2-81d8-4044-8ea8-a664383f888b/volumes"
Oct 30 03:17:43.979912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.979834    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:17:43.980653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980638    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:17:43.980716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980654    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:17:43.980716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980663    4214 cpu_manager.go:414] "containerMap: removing container" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:17:43.980716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980668    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388f6b73-1004-46b4-80a2-badc595341c8" containerName="cleanup"
Oct 30 03:17:43.980716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980692    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing"
Oct 30 03:17:43.980716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980698    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980720    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980725    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980739    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980744    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980751    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980756    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980770    4214 cpu_manager.go:414] "containerMap: removing container" podUID="57cd6b97-58e0-4803-9145-9673538a3546" containerName="maas-api"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980775    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cd6b97-58e0-4803-9145-9673538a3546" containerName="maas-api"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980784    4214 cpu_manager.go:414] "containerMap: removing container" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980788    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980798    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980802    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980811    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980816    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980826    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980831    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980837    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main"
Oct 30 03:17:43.980886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980841    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980944    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980952    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe71af4-84cc-4e82-a458-9eb83195d4d8" containerName="main"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980959    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf0b28-ef66-4a56-97ed-68b4254092b0" containerName="cleanup"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980969    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff44dcf2-81d8-4044-8ea8-a664383f888b" containerName="payload-pre-processing"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980977    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2139de-66e7-4906-8dfb-35015be69017" containerName="main"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980983    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbe6f81-793d-4a2f-be93-856cf3405fbf" containerName="istio-proxy"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980989    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9290e3e-cc98-4f16-b348-109b43f92e30" containerName="payload-processing"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.980997    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cd6b97-58e0-4803-9145-9673538a3546" containerName="maas-api"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.981002    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacbde12-13e2-4a6c-9320-2a7cf0b6e0cd" containerName="istio-proxy"
Oct 30 03:17:43.981578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.981008    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5350992a-dde7-47b0-9f06-d2c9acd77a76" containerName="cleanup"
Oct 30 03:17:43.987301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:43.987281    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.007821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.006156    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:17:44.009115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.009092    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4cf8f5\"/\"kube-root-ca.crt\""
Oct 30 03:17:44.009298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.009124    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4cf8f5\"/\"openshift-service-ca.crt\""
Oct 30 03:17:44.015611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.015593    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4cf8f5\"/\"default-dockercfg-f7znd\""
Oct 30 03:17:44.015717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.015681    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4cf8f5\"/\"test-model-4cf8f5-kserve-self-signed-certs\""
Oct 30 03:17:44.051264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051217    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.051434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051341    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.051573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051552    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.051642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051599    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.051785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051770    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwp2\" (UniqueName: \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.051847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.051827    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153310    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153356    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153477    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwp2\" (UniqueName: \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153520    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153700    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.153811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153770    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.154077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153816    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.154077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.153853    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.154077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.154034    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.156199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.156176    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.156451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.156430    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.173170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.173147    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwp2\" (UniqueName: \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\") pod \"test-model-4cf8f5-kserve-b77d66555-bqcvz\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") " pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:44.311768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:44.311731    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:45.373257 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:17:45.349352    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1730cd1a_43ff_46bd_ad97_847d0497ce8a.slice/crio-ec7fdd82bd2b3bf128f1708ebea8d25eebae59d3e19cbc0b4c0ad759948d1937 WatchSource:0}: Error finding container ec7fdd82bd2b3bf128f1708ebea8d25eebae59d3e19cbc0b4c0ad759948d1937: Status 404 returned error can't find the container with id ec7fdd82bd2b3bf128f1708ebea8d25eebae59d3e19cbc0b4c0ad759948d1937
Oct 30 03:17:45.428262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.425282    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:17:45.503360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.503313    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": context deadline exceeded"
Oct 30 03:17:45.589787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.507637    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:45.682253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.681487    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:45.727253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.726259    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:45.878256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:45.865378    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:17:46.087352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.020363    4214 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/health\": context deadline exceeded"
Oct 30 03:17:46.096380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.096322    4214 patch_prober.go:29] interesting pod/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:17:46.096593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.096404    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:46.147252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.144050    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:17:46.799589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.799542    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:46.815785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.815725    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:17:46.816490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:46.816466    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:17:47.379557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:47.379518    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" event={"ID":"1730cd1a-43ff-46bd-ad97-847d0497ce8a","Type":"ContainerStarted","Data":"ec7fdd82bd2b3bf128f1708ebea8d25eebae59d3e19cbc0b4c0ad759948d1937"}
Oct 30 03:17:47.463912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:47.463882    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:17:48.312299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:48.312265    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" event={"ID":"1730cd1a-43ff-46bd-ad97-847d0497ce8a","Type":"ContainerStarted","Data":"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"}
Oct 30 03:17:48.363172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:48.363110    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" podStartSLOduration=5.3630928319999995 podStartE2EDuration="5.363092832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:48.358264131 +0000 UTC m=+2207.432434921" watchObservedRunningTime="2026-10-30 03:17:48.363092832 +0000 UTC m=+2207.437263626"
Oct 30 03:17:49.312296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:49.312266    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:50.879522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:50.879482    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing" containerID="cri-o://4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768" gracePeriod=25
Oct 30 03:17:51.145360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:51.145276    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing" containerID="cri-o://f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405" gracePeriod=25
Oct 30 03:17:53.352395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:53.352360    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:17:54.312568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:54.312531    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:54.329002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:54.328969    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:54.398084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:54.398052    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:17:55.692873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:55.692828    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:17:56.703191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:17:56.703146    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:04.092983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.092945    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:18:04.101161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.101105    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:18:04.444243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.444144    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:18:04.445860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.445829    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c9d962-30dd-4ec5-9c07-0470736111d2" containerName="istio-proxy"
Oct 30 03:18:04.450575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.450552    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.478221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.478187    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:18:04.543703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.543659    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq89\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.543894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.543812    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.543994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.543978    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544013    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544126    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544158    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544247    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544285    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.544443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.544404    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646174    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646259    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq89\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646347    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646493    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646530    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646601    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646626    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646655    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.646857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646680    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.647058 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.646987    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.647190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.647077    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.647190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.647092    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.647514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.647489    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.647715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.647690    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.649003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.648980    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.649381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.649361    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.666396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.666363    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.666930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.666906    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq89\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:04.776595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:04.776513    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:05.215324 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:18:05.215294    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa4eeb6_9813_4196_b508_69d3943c4f73.slice/crio-b925439076de311569ef107c3c2189d5cbcf3dd669bb9dead4e4d6a06a108034 WatchSource:0}: Error finding container b925439076de311569ef107c3c2189d5cbcf3dd669bb9dead4e4d6a06a108034: Status 404 returned error can't find the container with id b925439076de311569ef107c3c2189d5cbcf3dd669bb9dead4e4d6a06a108034
Oct 30 03:18:05.216625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.216599    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:18:05.218724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.218691    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.218809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.218796    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.218856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.218827    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:05.532625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.532546    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" event={"ID":"caa4eeb6-9813-4196-b508-69d3943c4f73","Type":"ContainerStarted","Data":"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"}
Oct 30 03:18:05.532777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.532626    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" event={"ID":"caa4eeb6-9813-4196-b508-69d3943c4f73","Type":"ContainerStarted","Data":"b925439076de311569ef107c3c2189d5cbcf3dd669bb9dead4e4d6a06a108034"}
Oct 30 03:18:05.601982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.601933    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podStartSLOduration=1.601919433 podStartE2EDuration="1.601919433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:05.588724329 +0000 UTC m=+2224.662895124" watchObservedRunningTime="2026-10-30 03:18:05.601919433 +0000 UTC m=+2224.676090282"
Oct 30 03:18:05.692196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.692162    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:05.777144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:05.777113    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:06.702940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:06.702854    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:06.779283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:06.779244    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:06.779475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:06.779322    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:07.777528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:07.777485    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:07.777947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:07.777546    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:08.390099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.390063    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main" containerID="cri-o://d8888dbf0acfa4ad29122b6750d4cef5af1d5f4ae3e457e8e600c1b3336dc864" gracePeriod=45
Oct 30 03:18:08.595533 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.595496    4214 generic.go:356] "Generic (PLEG): container finished" podID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerID="d8888dbf0acfa4ad29122b6750d4cef5af1d5f4ae3e457e8e600c1b3336dc864" exitCode=0
Oct 30 03:18:08.595709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.595635    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" event={"ID":"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509","Type":"ContainerDied","Data":"d8888dbf0acfa4ad29122b6750d4cef5af1d5f4ae3e457e8e600c1b3336dc864"}
Oct 30 03:18:08.777698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.777665    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:08.778092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.777741    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:08.781837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.781816    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.868802    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.869022    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.869059    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.869140    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\" (UniqueName: \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.869216    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.869292    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\") pod \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\" (UID: \"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509\") "
Oct 30 03:18:08.871258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.870973    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:08.871684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.871467    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:08.872315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.872291    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:08.872416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.872318    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:08.874335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.874308    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:08.890631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.890585    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:08.891716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.891684    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:08.892769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:08.892743    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj" pod "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" (UID: "0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509"). InnerVolumeSpecName "kube-api-access-ws2kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:09.024276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.023747    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:09.024276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.023783    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:09.024276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.023798    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2kj\" (UniqueName: \"kubernetes.io/projected/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-kube-api-access-ws2kj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:09.024276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.023813    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:09.098330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.098287    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing" containerID="cri-o://7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5" gracePeriod=25
Oct 30 03:18:09.101952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.101832    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing" containerID="cri-o://b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1" gracePeriod=25
Oct 30 03:18:09.195991 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:09.195951    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:18:09.606686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.606655    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"
Oct 30 03:18:09.606686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.606677    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp" event={"ID":"0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509","Type":"ContainerDied","Data":"d26629a807e2b37b018f844b2e40a6669888f9bf8dbf1ce444ec4753a636f14a"}
Oct 30 03:18:09.606998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.606731    4214 scope.go:122] "RemoveContainer" containerID="d8888dbf0acfa4ad29122b6750d4cef5af1d5f4ae3e457e8e600c1b3336dc864"
Oct 30 03:18:09.683671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.681436    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:18:09.692500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.691808    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-537001/test-model-537001-kserve-778c6f658c-8p4jp"]
Oct 30 03:18:09.779099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.779058    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:09.779501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:09.779158    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:10.783463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:10.777614    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:10.783463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:10.777718    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:11.099348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.099311    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" path="/var/lib/kubelet/pods/0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509/volumes"
Oct 30 03:18:11.328014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.325352    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:18:11.422991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.422955    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\") pod \"6c237a67-034f-4fb7-8423-790f9269bf55\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") "
Oct 30 03:18:11.423212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.423198    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\" (UniqueName: \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\") pod \"6c237a67-034f-4fb7-8423-790f9269bf55\" (UID: \"6c237a67-034f-4fb7-8423-790f9269bf55\") "
Oct 30 03:18:11.424408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.424382    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume" pod "6c237a67-034f-4fb7-8423-790f9269bf55" (UID: "6c237a67-034f-4fb7-8423-790f9269bf55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:11.428020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.427997    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk" pod "6c237a67-034f-4fb7-8423-790f9269bf55" (UID: "6c237a67-034f-4fb7-8423-790f9269bf55"). InnerVolumeSpecName "kube-api-access-5m8zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:11.526988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.526945    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c237a67-034f-4fb7-8423-790f9269bf55-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:11.526988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.526980    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8zk\" (UniqueName: \"kubernetes.io/projected/6c237a67-034f-4fb7-8423-790f9269bf55-kube-api-access-5m8zk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:11.574304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.573592    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:18:11.627961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.627920    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\" (UniqueName: \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\") pod \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") "
Oct 30 03:18:11.628164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.628060    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\") pod \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\" (UID: \"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67\") "
Oct 30 03:18:11.628725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.628692    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume" pod "bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" (UID: "bd31f5ac-ee3d-4d70-b477-5c37d72bbb67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:11.629179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.629154    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:11.630626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630596    4214 generic.go:356] "Generic (PLEG): container finished" podID="6c237a67-034f-4fb7-8423-790f9269bf55" containerID="4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768" exitCode=0
Oct 30 03:18:11.630751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630655    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t" pod "bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" (UID: "bd31f5ac-ee3d-4d70-b477-5c37d72bbb67"). InnerVolumeSpecName "kube-api-access-phs8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:11.630751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630677    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"
Oct 30 03:18:11.630866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630765    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" event={"ID":"6c237a67-034f-4fb7-8423-790f9269bf55","Type":"ContainerDied","Data":"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"}
Oct 30 03:18:11.630866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630824    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7" event={"ID":"6c237a67-034f-4fb7-8423-790f9269bf55","Type":"ContainerDied","Data":"52e936dfa61e7cc306f636ff7563ab268c450770f598c88a467e7a7c4341eb6c"}
Oct 30 03:18:11.630866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.630859    4214 scope.go:122] "RemoveContainer" containerID="4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"
Oct 30 03:18:11.633086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.633064    4214 generic.go:356] "Generic (PLEG): container finished" podID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerID="f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405" exitCode=0
Oct 30 03:18:11.638277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.633122    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"
Oct 30 03:18:11.646644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.633139    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" event={"ID":"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67","Type":"ContainerDied","Data":"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"}
Oct 30 03:18:11.646801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.646781    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv" event={"ID":"bd31f5ac-ee3d-4d70-b477-5c37d72bbb67","Type":"ContainerDied","Data":"c9e23182605962fe2390f85cb7a339e98b00f4593d324a0f26313eba65d9b609"}
Oct 30 03:18:11.649826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.649810    4214 scope.go:122] "RemoveContainer" containerID="4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"
Oct 30 03:18:11.650394 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:11.650212    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768\": container with ID starting with 4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768 not found: ID does not exist" containerID="4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"
Oct 30 03:18:11.650394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.650290    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768"} err="failed to get container status \"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768\": rpc error: code = NotFound desc = could not find container \"4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768\": container with ID starting with 4b0e227039ee15e6d20e08432dc45c99b5ba4b64b337bd86fb8b8df6859fe768 not found: ID does not exist"
Oct 30 03:18:11.650394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.650306    4214 scope.go:122] "RemoveContainer" containerID="f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"
Oct 30 03:18:11.691002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.690961    4214 scope.go:122] "RemoveContainer" containerID="f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"
Oct 30 03:18:11.691742 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:11.691677    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405\": container with ID starting with f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405 not found: ID does not exist" containerID="f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"
Oct 30 03:18:11.691742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.691714    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405"} err="failed to get container status \"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405\": rpc error: code = NotFound desc = could not find container \"f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405\": container with ID starting with f437077d4d17de3a08ba1dc6a244c3b91d5d455e8ce6eb84431969b46527b405 not found: ID does not exist"
Oct 30 03:18:11.718282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.718249    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:18:11.723196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.723169    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4f415d-7bb64d8799-rpwsv"]
Oct 30 03:18:11.731117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.731086    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phs8t\" (UniqueName: \"kubernetes.io/projected/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67-kube-api-access-phs8t\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:11.759566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.759520    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:18:11.767070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.767035    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f415d-5d8f4cb6b-wsmw7"]
Oct 30 03:18:11.778469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.778439    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 03:18:11.778723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:11.778687    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 03:18:12.783580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:12.783548    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:12.784149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:12.784128    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:12.785325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:12.785298    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:18:13.071893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:13.071859    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" path="/var/lib/kubelet/pods/6c237a67-034f-4fb7-8423-790f9269bf55/volumes"
Oct 30 03:18:13.072381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:13.072366    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" path="/var/lib/kubelet/pods/bd31f5ac-ee3d-4d70-b477-5c37d72bbb67/volumes"
Oct 30 03:18:15.222372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.222333    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:15.222789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.222384    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:15.270211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.270176    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:18:15.272952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.270641    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" podUID="ed7e6cb1-a252-4add-aee1-b44344b8095d" containerName="maas-api" containerID="cri-o://71906b304185b720b5a0564f79eee59efba72b94fd853a4a317120d494c81c6f" gracePeriod=30
Oct 30 03:18:15.817254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.785591    4214 generic.go:356] "Generic (PLEG): container finished" podID="ed7e6cb1-a252-4add-aee1-b44344b8095d" containerID="71906b304185b720b5a0564f79eee59efba72b94fd853a4a317120d494c81c6f" exitCode=0
Oct 30 03:18:15.817254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:15.785928    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" event={"ID":"ed7e6cb1-a252-4add-aee1-b44344b8095d","Type":"ContainerDied","Data":"71906b304185b720b5a0564f79eee59efba72b94fd853a4a317120d494c81c6f"}
Oct 30 03:18:16.017276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.015380    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:18:16.089316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.089268    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\") pod \"ed7e6cb1-a252-4add-aee1-b44344b8095d\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") "
Oct 30 03:18:16.089480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.089371    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") pod \"ed7e6cb1-a252-4add-aee1-b44344b8095d\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") "
Oct 30 03:18:16.089674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.089654    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\" (UniqueName: \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\") pod \"ed7e6cb1-a252-4add-aee1-b44344b8095d\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") "
Oct 30 03:18:16.089822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.089789    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\") pod \"ed7e6cb1-a252-4add-aee1-b44344b8095d\" (UID: \"ed7e6cb1-a252-4add-aee1-b44344b8095d\") "
Oct 30 03:18:16.090192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.090165    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca" pod "ed7e6cb1-a252-4add-aee1-b44344b8095d" (UID: "ed7e6cb1-a252-4add-aee1-b44344b8095d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:16.090586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.090567    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7e6cb1-a252-4add-aee1-b44344b8095d-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:16.092028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.092003    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls" pod "ed7e6cb1-a252-4add-aee1-b44344b8095d" (UID: "ed7e6cb1-a252-4add-aee1-b44344b8095d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:16.092143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.092045    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4" pod "ed7e6cb1-a252-4add-aee1-b44344b8095d" (UID: "ed7e6cb1-a252-4add-aee1-b44344b8095d"). InnerVolumeSpecName "kube-api-access-l97r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:16.093183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.093154    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs" pod "ed7e6cb1-a252-4add-aee1-b44344b8095d" (UID: "ed7e6cb1-a252-4add-aee1-b44344b8095d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:16.191919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.191888    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97r4\" (UniqueName: \"kubernetes.io/projected/ed7e6cb1-a252-4add-aee1-b44344b8095d-kube-api-access-l97r4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:16.191919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.191916    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:16.191919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.191927    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7e6cb1-a252-4add-aee1-b44344b8095d-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:16.795762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.795732    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"
Oct 30 03:18:16.796323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.795769    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt" event={"ID":"ed7e6cb1-a252-4add-aee1-b44344b8095d","Type":"ContainerDied","Data":"b958c90fe4c22dc3c3e61ea347ed5381b4445435d1cfbee40ae64e3da896b77f"}
Oct 30 03:18:16.796323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.795835    4214 scope.go:122] "RemoveContainer" containerID="71906b304185b720b5a0564f79eee59efba72b94fd853a4a317120d494c81c6f"
Oct 30 03:18:16.852293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.852260    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:18:16.861575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:16.861542    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f415d-69746c4cc4-cltlt"]
Oct 30 03:18:17.070443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:17.070356    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7e6cb1-a252-4add-aee1-b44344b8095d" path="/var/lib/kubelet/pods/ed7e6cb1-a252-4add-aee1-b44344b8095d/volumes"
Oct 30 03:18:24.196829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:24.196764    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:18:24.197623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:24.197014    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" containerID="cri-o://07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3" gracePeriod=30
Oct 30 03:18:25.223672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.223627    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:25.224073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.223677    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:25.590877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.590842    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:18:25.591709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591690    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591711    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591738    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ed7e6cb1-a252-4add-aee1-b44344b8095d" containerName="maas-api"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591743    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7e6cb1-a252-4add-aee1-b44344b8095d" containerName="maas-api"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591755    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591759    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591780    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing"
Oct 30 03:18:25.591787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591786    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing"
Oct 30 03:18:25.592060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591928    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c237a67-034f-4fb7-8423-790f9269bf55" containerName="payload-processing"
Oct 30 03:18:25.592060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591946    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd31f5ac-ee3d-4d70-b477-5c37d72bbb67" containerName="payload-pre-processing"
Oct 30 03:18:25.592060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591963    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7e6cb1-a252-4add-aee1-b44344b8095d" containerName="maas-api"
Oct 30 03:18:25.592060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.591974    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a2d0a-b5c2-4d95-ae03-e5bed8c53509" containerName="main"
Oct 30 03:18:25.598068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.598042    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.629540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.629509    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-07e5c2-openshift-default-dockercfg-nrgpb\""
Oct 30 03:18:25.633677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.633650    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:18:25.673848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.673815    4214 patch_prober.go:29] interesting pod/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:18:25.674083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.674058    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:18:25.727301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.715513    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.757450    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762331    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762434    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762527    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762687    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlcx\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762725    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762869    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.773250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.762915    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.864122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.864032    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlcx\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.864396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.864373    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.878251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.864605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.880352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.880322    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.880890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.880871    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.881469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.881449    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.882007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.881988    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.882207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.882191    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.882435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.882414    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.900321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.880800    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.900321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.898435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.900321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.879162    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.901264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.901222    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.902178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.902153    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.917938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.917907    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.956250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.931860    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.956250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.936750    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:25.956250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:25.937359    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlcx\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\") pod \"e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") " pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:26.223332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:26.223225    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:26.552032 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:18:26.551998    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c6251_7198_4fba_be5c_e2504a0dee51.slice/crio-c3ebaf1b31ad25363dd9de38d99828058a4067dae10d7447ade2f02bc22c8df0 WatchSource:0}: Error finding container c3ebaf1b31ad25363dd9de38d99828058a4067dae10d7447ade2f02bc22c8df0: Status 404 returned error can't find the container with id c3ebaf1b31ad25363dd9de38d99828058a4067dae10d7447ade2f02bc22c8df0
Oct 30 03:18:26.552895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:26.552866    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:18:26.555940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:26.555898    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:26.556026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:26.555990    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:26.556078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:26.556036    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:27.115969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:27.115931    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerStarted","Data":"777084db5f9b9a18e15d87f6266b43aa27c334b666e9482e6766b0ddaa1e9097"}
Oct 30 03:18:27.116169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:27.115991    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerStarted","Data":"c3ebaf1b31ad25363dd9de38d99828058a4067dae10d7447ade2f02bc22c8df0"}
Oct 30 03:18:27.180160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:27.180095    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podStartSLOduration=2.180075602 podStartE2EDuration="2.180075602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:27.173568078 +0000 UTC m=+2246.247738882" watchObservedRunningTime="2026-10-30 03:18:27.180075602 +0000 UTC m=+2246.254246397"
Oct 30 03:18:27.224178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:27.224142    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:27.230329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:27.230305    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:28.124317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:28.124280    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:28.125372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:28.125347    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:18:29.431964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.431937    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:18:29.519312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.518894    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:18:29.556885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.556850    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\" (UniqueName: \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\") pod \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") "
Oct 30 03:18:29.557141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.557123    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\") pod \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\" (UID: \"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5\") "
Oct 30 03:18:29.557837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.557805    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume" pod "9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" (UID: "9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:29.564502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.564212    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd" pod "9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" (UID: "9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5"). InnerVolumeSpecName "kube-api-access-w5cqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.583678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.583652    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:18:29.658498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658405    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658502    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658552    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658599    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658655    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658716    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.658940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658714    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.658940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658813    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\" (UniqueName: \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\") pod \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") "
Oct 30 03:18:29.658940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658868    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.659243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.658950    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.659243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.659080    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\") pod \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\" (UID: \"bd00b2bf-87ec-44e2-9696-74fff694ab8c\") "
Oct 30 03:18:29.659243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.659123    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\") pod \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\" (UID: \"b3c40987-aae2-40e2-8ccc-d0552ad44e85\") "
Oct 30 03:18:29.659981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.659956    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.660114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.659985    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5cqd\" (UniqueName: \"kubernetes.io/projected/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-kube-api-access-w5cqd\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.660114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.660000    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.660330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.660264    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.660722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.660701    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.661298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.661002    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:29.661298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.661251    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.661882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.661855    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume" pod "bd00b2bf-87ec-44e2-9696-74fff694ab8c" (UID: "bd00b2bf-87ec-44e2-9696-74fff694ab8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:29.663969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.663946    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.665144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.665110    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:29.665298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.665195    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz" pod "bd00b2bf-87ec-44e2-9696-74fff694ab8c" (UID: "bd00b2bf-87ec-44e2-9696-74fff694ab8c"). InnerVolumeSpecName "kube-api-access-lx8qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.665491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.665466    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:29.666705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.666680    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz" pod "b3c40987-aae2-40e2-8ccc-d0552ad44e85" (UID: "b3c40987-aae2-40e2-8ccc-d0552ad44e85"). InnerVolumeSpecName "kube-api-access-m8hgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:29.761593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761553    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761594    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd00b2bf-87ec-44e2-9696-74fff694ab8c-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761609    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761624    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761638    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761653    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761666    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hgz\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-kube-api-access-m8hgz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761679    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761693    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8qz\" (UniqueName: \"kubernetes.io/projected/bd00b2bf-87ec-44e2-9696-74fff694ab8c-kube-api-access-lx8qz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:29.761868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:29.761706    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c40987-aae2-40e2-8ccc-d0552ad44e85-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:30.142807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.142770    4214 generic.go:356] "Generic (PLEG): container finished" podID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerID="7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5" exitCode=0
Oct 30 03:18:30.143017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.142820    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"
Oct 30 03:18:30.143017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.142857    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" event={"ID":"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5","Type":"ContainerDied","Data":"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"}
Oct 30 03:18:30.143017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.142911    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr" event={"ID":"9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5","Type":"ContainerDied","Data":"f1af7ba1ce4573e34f153f8649e653b4e98fb682ed1693b8d8c5e83971734a01"}
Oct 30 03:18:30.143017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.142942    4214 scope.go:122] "RemoveContainer" containerID="7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"
Oct 30 03:18:30.144908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.144884    4214 generic.go:356] "Generic (PLEG): container finished" podID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerID="b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1" exitCode=0
Oct 30 03:18:30.145050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.144929    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"
Oct 30 03:18:30.145050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.144967    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" event={"ID":"bd00b2bf-87ec-44e2-9696-74fff694ab8c","Type":"ContainerDied","Data":"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"}
Oct 30 03:18:30.145050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.145013    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb" event={"ID":"bd00b2bf-87ec-44e2-9696-74fff694ab8c","Type":"ContainerDied","Data":"70ce291b858c82e0c8adfacb88c73075882281b111e53c178a82a799c77dd964"}
Oct 30 03:18:30.146922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.146900    4214 generic.go:356] "Generic (PLEG): container finished" podID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerID="07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3" exitCode=0
Oct 30 03:18:30.147038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.146955    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"
Oct 30 03:18:30.147038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.147006    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" event={"ID":"b3c40987-aae2-40e2-8ccc-d0552ad44e85","Type":"ContainerDied","Data":"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"}
Oct 30 03:18:30.147175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.147044    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb" event={"ID":"b3c40987-aae2-40e2-8ccc-d0552ad44e85","Type":"ContainerDied","Data":"94b8a5c2901c1d7c7aa0fb14556514ea59f9d37f5490283698768d255dd84d69"}
Oct 30 03:18:30.158944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.158921    4214 scope.go:122] "RemoveContainer" containerID="7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"
Oct 30 03:18:30.159333 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:30.159302    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5\": container with ID starting with 7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5 not found: ID does not exist" containerID="7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"
Oct 30 03:18:30.159492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.159468    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5"} err="failed to get container status \"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5\": rpc error: code = NotFound desc = could not find container \"7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5\": container with ID starting with 7a004ab8dcf68479f0aeb444af8a2e4e6c4cfcfdfce54c83a6b088848ff3bfb5 not found: ID does not exist"
Oct 30 03:18:30.159593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.159574    4214 scope.go:122] "RemoveContainer" containerID="b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"
Oct 30 03:18:30.172704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.172680    4214 scope.go:122] "RemoveContainer" containerID="b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"
Oct 30 03:18:30.173020 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:30.173000    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1\": container with ID starting with b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1 not found: ID does not exist" containerID="b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"
Oct 30 03:18:30.173097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.173026    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1"} err="failed to get container status \"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1\": rpc error: code = NotFound desc = could not find container \"b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1\": container with ID starting with b4924d11c6f944b0eb300f0e6a0b77dff76f9f271df6ab52068892d801fa60f1 not found: ID does not exist"
Oct 30 03:18:30.173097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.173045    4214 scope.go:122] "RemoveContainer" containerID="07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"
Oct 30 03:18:30.184400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.184377    4214 scope.go:122] "RemoveContainer" containerID="07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"
Oct 30 03:18:30.184685 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:30.184661    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3\": container with ID starting with 07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3 not found: ID does not exist" containerID="07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"
Oct 30 03:18:30.184755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.184696    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3"} err="failed to get container status \"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3\": rpc error: code = NotFound desc = could not find container \"07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3\": container with ID starting with 07380a162c44834a930716c629a8ee4db921af5c2a2421660db99041991123a3 not found: ID does not exist"
Oct 30 03:18:30.222828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.222798    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:18:30.228877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.228851    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16df10-58fb5c86cb-5bhpr"]
Oct 30 03:18:30.282805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.282775    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:18:30.287534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.287512    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4f415d-openshift-default-549bd7f477-7z2qb"]
Oct 30 03:18:30.318884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.318848    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:18:30.324225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.324196    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16df10-859bf4d79-25wlb"]
Oct 30 03:18:30.378347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.375547    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:18:30.378347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:30.376458    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" podUID="63bedab7-7e5d-498e-9069-560e0c933704" containerName="maas-api" containerID="cri-o://31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5" gracePeriod=30
Oct 30 03:18:31.103470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.103268    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:18:31.108938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.108905    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" path="/var/lib/kubelet/pods/9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5/volumes"
Oct 30 03:18:31.109459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.109435    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" path="/var/lib/kubelet/pods/b3c40987-aae2-40e2-8ccc-d0552ad44e85/volumes"
Oct 30 03:18:31.109853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.109837    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" path="/var/lib/kubelet/pods/bd00b2bf-87ec-44e2-9696-74fff694ab8c/volumes"
Oct 30 03:18:31.164628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.164593    4214 generic.go:356] "Generic (PLEG): container finished" podID="63bedab7-7e5d-498e-9069-560e0c933704" containerID="31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5" exitCode=0
Oct 30 03:18:31.164802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.164650    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"
Oct 30 03:18:31.164802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.164681    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" event={"ID":"63bedab7-7e5d-498e-9069-560e0c933704","Type":"ContainerDied","Data":"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"}
Oct 30 03:18:31.164802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.164739    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" event={"ID":"63bedab7-7e5d-498e-9069-560e0c933704","Type":"ContainerDied","Data":"e10e3cdd95473dc85cf5af544adb2b7e695d2afbf5689363d33ea2f75d5c6bcd"}
Oct 30 03:18:31.164802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.164773    4214 scope.go:122] "RemoveContainer" containerID="31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"
Oct 30 03:18:31.180341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.180321    4214 scope.go:122] "RemoveContainer" containerID="31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"
Oct 30 03:18:31.180692 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:31.180665    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5\": container with ID starting with 31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5 not found: ID does not exist" containerID="31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"
Oct 30 03:18:31.180837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.180698    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5"} err="failed to get container status \"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5\": rpc error: code = NotFound desc = could not find container \"31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5\": container with ID starting with 31e277c6b3364d0335fd5620f334551a713eb1c55011b647e1c19dfab201b3f5 not found: ID does not exist"
Oct 30 03:18:31.298323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.298284    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\") pod \"63bedab7-7e5d-498e-9069-560e0c933704\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") "
Oct 30 03:18:31.298507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.298361    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\") pod \"63bedab7-7e5d-498e-9069-560e0c933704\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") "
Oct 30 03:18:31.298507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.298447    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") pod \"63bedab7-7e5d-498e-9069-560e0c933704\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") "
Oct 30 03:18:31.298656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.298525    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\" (UniqueName: \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\") pod \"63bedab7-7e5d-498e-9069-560e0c933704\" (UID: \"63bedab7-7e5d-498e-9069-560e0c933704\") "
Oct 30 03:18:31.298773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.298749    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca" pod "63bedab7-7e5d-498e-9069-560e0c933704" (UID: "63bedab7-7e5d-498e-9069-560e0c933704"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:31.299267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.299214    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bedab7-7e5d-498e-9069-560e0c933704-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:31.300634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.300610    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs" pod "63bedab7-7e5d-498e-9069-560e0c933704" (UID: "63bedab7-7e5d-498e-9069-560e0c933704"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:31.300738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.300630    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls" pod "63bedab7-7e5d-498e-9069-560e0c933704" (UID: "63bedab7-7e5d-498e-9069-560e0c933704"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:31.301199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.301177    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th" pod "63bedab7-7e5d-498e-9069-560e0c933704" (UID: "63bedab7-7e5d-498e-9069-560e0c933704"). InnerVolumeSpecName "kube-api-access-xj9th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:31.400249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.400160    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:31.400249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.400191    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9th\" (UniqueName: \"kubernetes.io/projected/63bedab7-7e5d-498e-9069-560e0c933704-kube-api-access-xj9th\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:31.400249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.400203    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63bedab7-7e5d-498e-9069-560e0c933704-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:31.544026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.543995    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:18:31.559434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.559401    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg"]
Oct 30 03:18:31.715782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.715671    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:18:31.721005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.720969    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:18:31.721933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.721909    4214 cpu_manager.go:414] "containerMap: removing container" podUID="63bedab7-7e5d-498e-9069-560e0c933704" containerName="maas-api"
Oct 30 03:18:31.721933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.721934    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bedab7-7e5d-498e-9069-560e0c933704" containerName="maas-api"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722010    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722021    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722042    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722050    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722069    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy"
Oct 30 03:18:31.722144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722077    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy"
Oct 30 03:18:31.723100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722295    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd00b2bf-87ec-44e2-9696-74fff694ab8c" containerName="payload-pre-processing"
Oct 30 03:18:31.723100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722318    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7b846a-b9a9-455e-8fa5-ed3b2f441ed5" containerName="payload-processing"
Oct 30 03:18:31.723100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722332    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c40987-aae2-40e2-8ccc-d0552ad44e85" containerName="istio-proxy"
Oct 30 03:18:31.723100 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.722346    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bedab7-7e5d-498e-9069-560e0c933704" containerName="maas-api"
Oct 30 03:18:31.727805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.727785    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.743610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.743584    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07e5c2\""
Oct 30 03:18:31.744149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.744128    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07e5c2-dockercfg-chvm9\""
Oct 30 03:18:31.747901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.747877    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:18:31.807475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.807438    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x826z\" (UniqueName: \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.807668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.807499    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.830356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.823735    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:18:31.856266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.849163    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:31.910748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.910710    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:31.911054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.911031    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x826z\" (UniqueName: \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.911180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.911165    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.911378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.911360    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwgm\" (UniqueName: \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:31.920955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.920927    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:31.937611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.937584    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:18:31.989812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:31.989719    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x826z\" (UniqueName: \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:32.013217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.013181    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:32.013696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.013675    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwgm\" (UniqueName: \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:32.015593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.015571    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:32.094765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.076996    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:32.165302 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.156216    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwgm\" (UniqueName: \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\") pod \"payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:32.190926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.190619    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:32.735775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.735734    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:18:32.741622 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:18:32.741591    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1730013_6ae5_4703_843a_063439e5c4ec.slice/crio-7c17e5fdcec289f60a590874d670e62e875e4af202830632299c11483ce56c0f WatchSource:0}: Error finding container 7c17e5fdcec289f60a590874d670e62e875e4af202830632299c11483ce56c0f: Status 404 returned error can't find the container with id 7c17e5fdcec289f60a590874d670e62e875e4af202830632299c11483ce56c0f
Oct 30 03:18:32.769064 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:32.768956    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:18:33.070809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.070773    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bedab7-7e5d-498e-9069-560e0c933704" path="/var/lib/kubelet/pods/63bedab7-7e5d-498e-9069-560e0c933704/volumes"
Oct 30 03:18:33.198122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.198070    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" event={"ID":"31a56217-364b-4a4c-8f8c-a3dadeb4eddd","Type":"ContainerStarted","Data":"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"}
Oct 30 03:18:33.198626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.198146    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" event={"ID":"31a56217-364b-4a4c-8f8c-a3dadeb4eddd","Type":"ContainerStarted","Data":"fad4a30d530d76e137f1a06ee5857b6a5811daeb7dfc1a616b9db6c1dacee32f"}
Oct 30 03:18:33.198626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.198176    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:33.199994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.199961    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" event={"ID":"c1730013-6ae5-4703-843a-063439e5c4ec","Type":"ContainerStarted","Data":"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"}
Oct 30 03:18:33.200124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.200016    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" event={"ID":"c1730013-6ae5-4703-843a-063439e5c4ec","Type":"ContainerStarted","Data":"7c17e5fdcec289f60a590874d670e62e875e4af202830632299c11483ce56c0f"}
Oct 30 03:18:33.200124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.200049    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:33.244957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.242147    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" podStartSLOduration=2.242127697 podStartE2EDuration="2.242127697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:33.238005786 +0000 UTC m=+2252.312176625" watchObservedRunningTime="2026-10-30 03:18:33.242127697 +0000 UTC m=+2252.316298491"
Oct 30 03:18:33.280976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.280920    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" podStartSLOduration=2.28090342 podStartE2EDuration="2.28090342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:33.276829612 +0000 UTC m=+2252.351000406" watchObservedRunningTime="2026-10-30 03:18:33.28090342 +0000 UTC m=+2252.355074213"
Oct 30 03:18:33.779507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:33.779457    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16df10-64f95587df-72bxg" podUID="63bedab7-7e5d-498e-9069-560e0c933704" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.206:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:18:38.548116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:38.545536    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:18:38.548116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:38.545809    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy" containerID="cri-o://9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8" gracePeriod=30
Oct 30 03:18:39.168796 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:39.168755    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:18:43.151836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.151802    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:18:43.158153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.158130    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.188107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.188069    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9d3552-openshift-default-dockercfg-k8rl8\""
Oct 30 03:18:43.199571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.196521    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:18:43.255298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255258    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255385    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255436    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255474    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255585    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255619    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255807    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255848    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5msn\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.255975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.255933    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.357775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.357734    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.357953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.357783    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5msn\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.357953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.357921    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358131    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358206    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358253    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358288    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358296    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358204    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358352    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.358512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358391    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.359026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358808    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.359026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358856    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.359026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.358991    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.360490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.360465    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.361132 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.361111    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.385548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.385518    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5msn\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.385698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.385678    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.473557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.473458    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:43.852588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.852056    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:18:43.864016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.863365    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:43.864016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.863459    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:43.864016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.863500    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:43.930757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:43.930733    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:18:44.082815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.082769    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.082824    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.082913    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083055 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.082952    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083168    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:44.083269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083173    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083343    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083414    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083501    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\") pod \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\" (UID: \"19c87460-d5f1-4a61-9d2d-d5fa3d836024\") "
Oct 30 03:18:44.083936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083820    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:44.084080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.083933    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:44.084156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084135    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:44.084570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084435    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.084570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084461    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.084570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084475    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.084570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084491    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.084852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.084612    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:44.085660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.085630    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "kube-api-access-vk6vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:44.086374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.086345    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:44.086618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.086595    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:44.086906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.086879    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token" pod "19c87460-d5f1-4a61-9d2d-d5fa3d836024" (UID: "19c87460-d5f1-4a61-9d2d-d5fa3d836024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:44.186449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.186407    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.186845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.186479    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6vh\" (UniqueName: \"kubernetes.io/projected/19c87460-d5f1-4a61-9d2d-d5fa3d836024-kube-api-access-vk6vh\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.186845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.186496    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.186845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.186512    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.186845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.186527    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c87460-d5f1-4a61-9d2d-d5fa3d836024-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:44.211550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.211517    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:18:44.211700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.211597    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:18:44.374576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.374492    4214 generic.go:356] "Generic (PLEG): container finished" podID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerID="9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8" exitCode=0
Oct 30 03:18:44.374786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.374764    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerDied","Data":"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"}
Oct 30 03:18:44.374870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.374829    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r" event={"ID":"19c87460-d5f1-4a61-9d2d-d5fa3d836024","Type":"ContainerDied","Data":"aeee982586d97241d5845974bf6e1e6811f93b7d69ec9330818c1e083f98d138"}
Oct 30 03:18:44.374870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.374864    4214 scope.go:122] "RemoveContainer" containerID="9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"
Oct 30 03:18:44.375114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.375057    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"
Oct 30 03:18:44.386779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.386186    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" event={"ID":"44c4af4c-7c45-440f-88b8-066a5020071d","Type":"ContainerStarted","Data":"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"}
Oct 30 03:18:44.386779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.386310    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" event={"ID":"44c4af4c-7c45-440f-88b8-066a5020071d","Type":"ContainerStarted","Data":"01bf6fa696dbf948c106e2aa7fa33a7e5be5d8276d890654388e211a72a9c745"}
Oct 30 03:18:44.399274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.399249    4214 scope.go:122] "RemoveContainer" containerID="c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"
Oct 30 03:18:44.413777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.413752    4214 scope.go:122] "RemoveContainer" containerID="9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"
Oct 30 03:18:44.414122 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:44.414093    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8\": container with ID starting with 9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8 not found: ID does not exist" containerID="9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"
Oct 30 03:18:44.414248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.414127    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8"} err="failed to get container status \"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8\": rpc error: code = NotFound desc = could not find container \"9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8\": container with ID starting with 9fe6f18d55bd7b7f9a7477cc81dcc45f90159859f84fbfa4734cabeb7e028ff8 not found: ID does not exist"
Oct 30 03:18:44.414248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.414146    4214 scope.go:122] "RemoveContainer" containerID="c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"
Oct 30 03:18:44.414448 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:44.414418    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e\": container with ID starting with c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e not found: ID does not exist" containerID="c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"
Oct 30 03:18:44.414517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.414458    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e"} err="failed to get container status \"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e\": rpc error: code = NotFound desc = could not find container \"c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e\": container with ID starting with c2960d4550c8c8a5badc788e6732bc2d7849aec67b2de00a3009d97f2eaa637e not found: ID does not exist"
Oct 30 03:18:44.474094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.474057    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:44.475752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.475716    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:44.475884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.475774    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:44.523822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.523769    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podStartSLOduration=1.523754094 podStartE2EDuration="1.523754094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:44.522884223 +0000 UTC m=+2263.597055017" watchObservedRunningTime="2026-10-30 03:18:44.523754094 +0000 UTC m=+2263.597924886"
Oct 30 03:18:44.573560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.573527    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:18:44.579186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:44.579157    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-16df10-openshift-default-6d7dffd555-4rf8r"]
Oct 30 03:18:45.074955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:45.074922    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" path="/var/lib/kubelet/pods/19c87460-d5f1-4a61-9d2d-d5fa3d836024/volumes"
Oct 30 03:18:45.474983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:45.474723    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:45.474983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:45.474783    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:46.476250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:46.475596    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:46.476250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:46.475653    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:46.761311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:46.757414    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main" containerID="cri-o://b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea" gracePeriod=45
Oct 30 03:18:47.184702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.184671    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:18:47.266319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266286    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\" (UniqueName: \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.266521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266428    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.266591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266562    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.266650 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266603    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.266705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266675    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.266838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.266815    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\" (UniqueName: \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\") pod \"f62b93e7-a3b6-462c-876b-470fe471d080\" (UID: \"f62b93e7-a3b6-462c-876b-470fe471d080\") "
Oct 30 03:18:47.267166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.267138    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.267348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.267308    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.267618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.267598    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.267703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.267626    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.267907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.267884    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.270030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.269996    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:47.270924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.270898    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:47.271448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.271423    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh" pod "f62b93e7-a3b6-462c-876b-470fe471d080" (UID: "f62b93e7-a3b6-462c-876b-470fe471d080"). InnerVolumeSpecName "kube-api-access-2qnxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:47.368961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.368879    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62b93e7-a3b6-462c-876b-470fe471d080-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.368961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.368909    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.368961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.368918    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62b93e7-a3b6-462c-876b-470fe471d080-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.368961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.368926    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnxh\" (UniqueName: \"kubernetes.io/projected/f62b93e7-a3b6-462c-876b-470fe471d080-kube-api-access-2qnxh\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:18:47.514914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.514868    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:47.515462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.514928    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:47.529254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.529200    4214 generic.go:356] "Generic (PLEG): container finished" podID="f62b93e7-a3b6-462c-876b-470fe471d080" containerID="b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea" exitCode=0
Oct 30 03:18:47.529863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.529753    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" event={"ID":"f62b93e7-a3b6-462c-876b-470fe471d080","Type":"ContainerDied","Data":"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"}
Oct 30 03:18:47.529863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.529819    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv" event={"ID":"f62b93e7-a3b6-462c-876b-470fe471d080","Type":"ContainerDied","Data":"8c9d8abfbec93d3c1ff29521f8ac01c49ee04ce71b5c6365b4a53a5a3dd52b9b"}
Oct 30 03:18:47.530141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.529843    4214 scope.go:122] "RemoveContainer" containerID="b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"
Oct 30 03:18:47.530419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.530359    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"
Oct 30 03:18:47.599219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.598862    4214 scope.go:122] "RemoveContainer" containerID="b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"
Oct 30 03:18:47.599376 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:18:47.599312    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea\": container with ID starting with b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea not found: ID does not exist" containerID="b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"
Oct 30 03:18:47.599376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.599358    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea"} err="failed to get container status \"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea\": rpc error: code = NotFound desc = could not find container \"b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea\": container with ID starting with b4248679eb0b1dae216171f0c72edf2a7d9a8c74b7df978bacaec9c10dc91eea not found: ID does not exist"
Oct 30 03:18:47.626418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.626331    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:18:47.631092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:47.631058    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e7f767/test-model-e7f767-kserve-b87db6dc9-m6knv"]
Oct 30 03:18:48.474845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:48.474811    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:48.475008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:48.474866    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:49.082473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:49.082308    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" path="/var/lib/kubelet/pods/f62b93e7-a3b6-462c-876b-470fe471d080/volumes"
Oct 30 03:18:49.477311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:49.474509    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:49.477311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:49.474583    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:49.605836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:49.605802    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:18:50.476352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:50.474730    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:18:50.476352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:50.474833    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:18:51.479252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:51.479202    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:51.479793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:51.479770    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:51.480444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:51.480423    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:18:54.607068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:54.606999    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing" containerID="cri-o://735a5f7c6e1ff1367e176d4e4285c6a740d34cbe073b2aee9d24ae2f5fdc6e44" gracePeriod=25
Oct 30 03:18:59.931881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:18:59.931787    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:01.647798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:01.647763    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:19:08.928683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.928640    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:19:08.929479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929461    4214 cpu_manager.go:414] "containerMap: removing container" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.929532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929480    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.929532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929511    4214 cpu_manager.go:414] "containerMap: removing container" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.929532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929516    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.929637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929544    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main"
Oct 30 03:19:08.929637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929551    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main"
Oct 30 03:19:08.929704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929665    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62b93e7-a3b6-462c-876b-470fe471d080" containerName="main"
Oct 30 03:19:08.929704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929676    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.929704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.929686    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c87460-d5f1-4a61-9d2d-d5fa3d836024" containerName="istio-proxy"
Oct 30 03:19:08.935666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.935642    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:08.959344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.959309    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-07e5c2-serving-cert\""
Oct 30 03:19:08.962047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:08.962005    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:19:09.101020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.100981    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfb8k\" (UniqueName: \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.101212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.101138    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.101212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.101171    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.101396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.101373    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.210704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.210616    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.211151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.211122    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfb8k\" (UniqueName: \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.219576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.219539    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.219833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.219804    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.241057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.241023    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.242253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.242213    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.252467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.247951    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.253390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.253362    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfb8k\" (UniqueName: \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\") pod \"maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.257583 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:09.257519    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:19:09.264166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.264131    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:09.587360 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:09.587322    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a86ecbd_0a65_4736_9b70_669c92109124.slice/crio-504c7bbf8ff76320f062a8a55c19995fad864e999f7c6deba22560dd98726b9d WatchSource:0}: Error finding container 504c7bbf8ff76320f062a8a55c19995fad864e999f7c6deba22560dd98726b9d: Status 404 returned error can't find the container with id 504c7bbf8ff76320f062a8a55c19995fad864e999f7c6deba22560dd98726b9d
Oct 30 03:19:09.588755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.588725    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:19:09.588879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.588862    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:19:09.840410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.840315    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" event={"ID":"1a86ecbd-0a65-4736-9b70-669c92109124","Type":"ContainerStarted","Data":"504c7bbf8ff76320f062a8a55c19995fad864e999f7c6deba22560dd98726b9d"}
Oct 30 03:19:09.936042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:09.935978    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:10.852534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:10.852492    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" event={"ID":"1a86ecbd-0a65-4736-9b70-669c92109124","Type":"ContainerStarted","Data":"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"}
Oct 30 03:19:10.852750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:10.852567    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:10.910258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:10.901869    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" podStartSLOduration=2.408793737 podStartE2EDuration="2.901850064s" totalImagesPullingTime="493.056327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:10.898350712 +0000 UTC m=+2289.972521507" watchObservedRunningTime="2026-10-30 03:19:10.901850064 +0000 UTC m=+2289.976020860"
Oct 30 03:19:12.370190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:12.370150    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:19:12.405423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:12.405386    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:19:14.938851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:14.938680    4214 generic.go:356] "Generic (PLEG): container finished" podID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerID="735a5f7c6e1ff1367e176d4e4285c6a740d34cbe073b2aee9d24ae2f5fdc6e44" exitCode=0
Oct 30 03:19:14.938851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:14.938735    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" event={"ID":"41e6a7af-8b52-4d61-88b3-a930ea49daf0","Type":"ContainerDied","Data":"735a5f7c6e1ff1367e176d4e4285c6a740d34cbe073b2aee9d24ae2f5fdc6e44"}
Oct 30 03:19:15.210663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.210584    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:19:15.290850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.290808    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\") pod \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") "
Oct 30 03:19:15.292076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.292039    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\" (UniqueName: \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\") pod \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\" (UID: \"41e6a7af-8b52-4d61-88b3-a930ea49daf0\") "
Oct 30 03:19:15.295811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.291662    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume" pod "41e6a7af-8b52-4d61-88b3-a930ea49daf0" (UID: "41e6a7af-8b52-4d61-88b3-a930ea49daf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:15.304742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.304708    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm" pod "41e6a7af-8b52-4d61-88b3-a930ea49daf0" (UID: "41e6a7af-8b52-4d61-88b3-a930ea49daf0"). InnerVolumeSpecName "kube-api-access-9xgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:15.397428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.397377    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgjm\" (UniqueName: \"kubernetes.io/projected/41e6a7af-8b52-4d61-88b3-a930ea49daf0-kube-api-access-9xgjm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:15.397633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.397435    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e6a7af-8b52-4d61-88b3-a930ea49daf0-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:15.683106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.683070    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:19:15.688874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:15.683567    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" podUID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" containerName="maas-api" containerID="cri-o://7b12f47307eb642111b267db44316d305af68a55aff9c26fde4cf19ec87840a6" gracePeriod=30
Oct 30 03:19:16.273380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.260622    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c" event={"ID":"41e6a7af-8b52-4d61-88b3-a930ea49daf0","Type":"ContainerDied","Data":"c3fc06a5f6642b7632770da505775f2deb808bd4bdb16f4a8a668b791b67a64e"}
Oct 30 03:19:16.273380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.260718    4214 scope.go:122] "RemoveContainer" containerID="735a5f7c6e1ff1367e176d4e4285c6a740d34cbe073b2aee9d24ae2f5fdc6e44"
Oct 30 03:19:16.273380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.271380    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"
Oct 30 03:19:16.277053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.276974    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:19:16.278308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.278141    4214 generic.go:356] "Generic (PLEG): container finished" podID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" containerID="7b12f47307eb642111b267db44316d305af68a55aff9c26fde4cf19ec87840a6" exitCode=0
Oct 30 03:19:16.278541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.278471    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" event={"ID":"e0f445be-95b2-4059-8d5f-a06d13f5a84f","Type":"ContainerDied","Data":"7b12f47307eb642111b267db44316d305af68a55aff9c26fde4cf19ec87840a6"}
Oct 30 03:19:16.365249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.364656    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\") pod \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") "
Oct 30 03:19:16.365249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.364724    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\") pod \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") "
Oct 30 03:19:16.365249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.364837    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\") pod \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") "
Oct 30 03:19:16.365249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.365073    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\" (UniqueName: \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\") pod \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\" (UID: \"e0f445be-95b2-4059-8d5f-a06d13f5a84f\") "
Oct 30 03:19:16.397254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.397103    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca" pod "e0f445be-95b2-4059-8d5f-a06d13f5a84f" (UID: "e0f445be-95b2-4059-8d5f-a06d13f5a84f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:16.421250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.397697    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2" pod "e0f445be-95b2-4059-8d5f-a06d13f5a84f" (UID: "e0f445be-95b2-4059-8d5f-a06d13f5a84f"). InnerVolumeSpecName "kube-api-access-44gf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:16.421250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.400690    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls" pod "e0f445be-95b2-4059-8d5f-a06d13f5a84f" (UID: "e0f445be-95b2-4059-8d5f-a06d13f5a84f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:16.421250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.403351    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs" pod "e0f445be-95b2-4059-8d5f-a06d13f5a84f" (UID: "e0f445be-95b2-4059-8d5f-a06d13f5a84f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:16.421250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.410214    4214 scope.go:122] "RemoveContainer" containerID="7b12f47307eb642111b267db44316d305af68a55aff9c26fde4cf19ec87840a6"
Oct 30 03:19:16.438255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.436158    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:19:16.486253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.480788    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:16.486253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.480840    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f445be-95b2-4059-8d5f-a06d13f5a84f-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:16.486253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.480861    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f445be-95b2-4059-8d5f-a06d13f5a84f-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:16.486253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.480878    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gf2\" (UniqueName: \"kubernetes.io/projected/e0f445be-95b2-4059-8d5f-a06d13f5a84f-kube-api-access-44gf2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:16.604218 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.578978    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537001-554c9c6887-rnj4c"]
Oct 30 03:19:16.680256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.679571    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main" containerID="cri-o://fb261a3baa9c379da96485db69b09ceaeb6b395df05ff9c75e34c95cec808c2d" gracePeriod=45
Oct 30 03:19:16.874307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:16.873784    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:19:17.107919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.107889    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" path="/var/lib/kubelet/pods/41e6a7af-8b52-4d61-88b3-a930ea49daf0/volumes"
Oct 30 03:19:17.290409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.290375    4214 generic.go:356] "Generic (PLEG): container finished" podID="84dd794b-07e2-443e-880a-8e1d0360334a" containerID="fb261a3baa9c379da96485db69b09ceaeb6b395df05ff9c75e34c95cec808c2d" exitCode=0
Oct 30 03:19:17.290818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.290445    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" event={"ID":"84dd794b-07e2-443e-880a-8e1d0360334a","Type":"ContainerDied","Data":"fb261a3baa9c379da96485db69b09ceaeb6b395df05ff9c75e34c95cec808c2d"}
Oct 30 03:19:17.291894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.291868    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"
Oct 30 03:19:17.292067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.291866    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9" event={"ID":"e0f445be-95b2-4059-8d5f-a06d13f5a84f","Type":"ContainerDied","Data":"a032491eb369cc6a551189c5a63f42a6aff380ff5c19091a2e12dc51a762c566"}
Oct 30 03:19:17.330794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.330767    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:19:17.371488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.371452    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" podUID="0be1c6c8-b965-46f7-b651-755cc14dcff4" containerName="payload-processing" containerID="cri-o://665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e" gracePeriod=25
Oct 30 03:19:17.371956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.371936    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:19:17.379057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.379033    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537001-5d7df6d4c8-7rjn9"]
Oct 30 03:19:17.405919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.405885    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing" containerID="cri-o://f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53" gracePeriod=25
Oct 30 03:19:17.411593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.411564    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.411719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.411622    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.411719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.411674    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.411840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.411786    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.412203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.411930    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:17.412203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.412000    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.412203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.412183    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\" (UniqueName: \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\") pod \"84dd794b-07e2-443e-880a-8e1d0360334a\" (UID: \"84dd794b-07e2-443e-880a-8e1d0360334a\") "
Oct 30 03:19:17.412498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.412277    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:17.412570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.412495    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:17.414254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.414211    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.414389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.414274    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.414389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.414291    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.414736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.414713    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:17.415356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.415329    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "kube-api-access-85xg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:17.415689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.415670    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm" pod "84dd794b-07e2-443e-880a-8e1d0360334a" (UID: "84dd794b-07e2-443e-880a-8e1d0360334a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:17.516192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.516158    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd794b-07e2-443e-880a-8e1d0360334a-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.516192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.516191    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd794b-07e2-443e-880a-8e1d0360334a-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.516415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.516205    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85xg5\" (UniqueName: \"kubernetes.io/projected/84dd794b-07e2-443e-880a-8e1d0360334a-kube-api-access-85xg5\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.636643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.636617    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:19:17.719114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.719084    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\") pod \"0be1c6c8-b965-46f7-b651-755cc14dcff4\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") "
Oct 30 03:19:17.719376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.719245    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\" (UniqueName: \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\") pod \"0be1c6c8-b965-46f7-b651-755cc14dcff4\" (UID: \"0be1c6c8-b965-46f7-b651-755cc14dcff4\") "
Oct 30 03:19:17.720307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.720276    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume" pod "0be1c6c8-b965-46f7-b651-755cc14dcff4" (UID: "0be1c6c8-b965-46f7-b651-755cc14dcff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:17.721873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.721842    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g" pod "0be1c6c8-b965-46f7-b651-755cc14dcff4" (UID: "0be1c6c8-b965-46f7-b651-755cc14dcff4"). InnerVolumeSpecName "kube-api-access-r6d2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:17.821896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.821860    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be1c6c8-b965-46f7-b651-755cc14dcff4-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:17.821896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:17.821893    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6d2g\" (UniqueName: \"kubernetes.io/projected/0be1c6c8-b965-46f7-b651-755cc14dcff4-kube-api-access-r6d2g\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:18.154599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.154545    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:19:18.155540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155515    4214 cpu_manager.go:414] "containerMap: removing container" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main"
Oct 30 03:19:18.155540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155539    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155556    4214 cpu_manager.go:414] "containerMap: removing container" podUID="0be1c6c8-b965-46f7-b651-755cc14dcff4" containerName="payload-processing"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155563    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be1c6c8-b965-46f7-b651-755cc14dcff4" containerName="payload-processing"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155595    4214 cpu_manager.go:414] "containerMap: removing container" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155603    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155625    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" containerName="maas-api"
Oct 30 03:19:18.155750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155630    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" containerName="maas-api"
Oct 30 03:19:18.156337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155781    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e6a7af-8b52-4d61-88b3-a930ea49daf0" containerName="payload-pre-processing"
Oct 30 03:19:18.156337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155794    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" containerName="maas-api"
Oct 30 03:19:18.156337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155806    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" containerName="main"
Oct 30 03:19:18.156337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.155813    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be1c6c8-b965-46f7-b651-755cc14dcff4" containerName="payload-processing"
Oct 30 03:19:18.160031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.160006    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.184307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.184271    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:19:18.188464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.188309    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"default-dockercfg-ml4hg\""
Oct 30 03:19:18.188464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.188425    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"rate-test-model-9245d1-kserve-self-signed-certs\""
Oct 30 03:19:18.188822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.188795    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"kube-root-ca.crt\""
Oct 30 03:19:18.188916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.188850    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9245d1\"/\"openshift-service-ca.crt\""
Oct 30 03:19:18.228505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228465    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.228720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228527    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.228720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228559    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.228856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228792    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vxq\" (UniqueName: \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.228918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228896    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.228974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.228951    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.305507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.305464    4214 generic.go:356] "Generic (PLEG): container finished" podID="0be1c6c8-b965-46f7-b651-755cc14dcff4" containerID="665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e" exitCode=0
Oct 30 03:19:18.305507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.305517    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"
Oct 30 03:19:18.305983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.305539    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" event={"ID":"0be1c6c8-b965-46f7-b651-755cc14dcff4","Type":"ContainerDied","Data":"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"}
Oct 30 03:19:18.305983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.305618    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2" event={"ID":"0be1c6c8-b965-46f7-b651-755cc14dcff4","Type":"ContainerDied","Data":"eb9e69484504372af7bf6b4e74b3fe7c30b23078f04259c17c3e26e87f184c74"}
Oct 30 03:19:18.305983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.305641    4214 scope.go:122] "RemoveContainer" containerID="665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"
Oct 30 03:19:18.307796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.307771    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r" event={"ID":"84dd794b-07e2-443e-880a-8e1d0360334a","Type":"ContainerDied","Data":"eb92cfedc5b5e75667a571e4d01f6c329a2ebc9d785061a8ce0c171164f64d66"}
Oct 30 03:19:18.307875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.307805    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"
Oct 30 03:19:18.319206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.319183    4214 scope.go:122] "RemoveContainer" containerID="665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"
Oct 30 03:19:18.319742 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:18.319722    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e\": container with ID starting with 665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e not found: ID does not exist" containerID="665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"
Oct 30 03:19:18.319876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.319846    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e"} err="failed to get container status \"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e\": rpc error: code = NotFound desc = could not find container \"665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e\": container with ID starting with 665c0e636230095120c7de5caa98df40e70f8adf7d917d3d815d26909802377e not found: ID does not exist"
Oct 30 03:19:18.319966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.319957    4214 scope.go:122] "RemoveContainer" containerID="fb261a3baa9c379da96485db69b09ceaeb6b395df05ff9c75e34c95cec808c2d"
Oct 30 03:19:18.330263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330218    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vxq\" (UniqueName: \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330342    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330383    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330453    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330490    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330518    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330761    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.330895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.330810    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.331152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.331117    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.333021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.332995    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.333116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.333063    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.351136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.351107    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vxq\" (UniqueName: \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\") pod \"rate-test-model-9245d1-kserve-54f9f9b745-j2tpp\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") " pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.367193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.367166    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:19:18.371713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.371688    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-100ffd/test-model-100ffd-kserve-7bfbb5bdf-jgj7r"]
Oct 30 03:19:18.472301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.457725    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:19:18.495252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.491532    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:18.495252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.494073    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7f767-844d6b5bdd-zlzl2"]
Oct 30 03:19:18.908716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:18.908677    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:19:18.908716 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:18.908691    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbef7941a_6f8b_45ec_b2cb_5e2b892e0164.slice/crio-d10be56b0261b9c245a4b34ccf7af11aebbc6c0fe34032b130a6832ef9a6a1e3 WatchSource:0}: Error finding container d10be56b0261b9c245a4b34ccf7af11aebbc6c0fe34032b130a6832ef9a6a1e3: Status 404 returned error can't find the container with id d10be56b0261b9c245a4b34ccf7af11aebbc6c0fe34032b130a6832ef9a6a1e3
Oct 30 03:19:19.070562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.070530    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be1c6c8-b965-46f7-b651-755cc14dcff4" path="/var/lib/kubelet/pods/0be1c6c8-b965-46f7-b651-755cc14dcff4/volumes"
Oct 30 03:19:19.071096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.071080    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dd794b-07e2-443e-880a-8e1d0360334a" path="/var/lib/kubelet/pods/84dd794b-07e2-443e-880a-8e1d0360334a/volumes"
Oct 30 03:19:19.071705 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.071687    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f445be-95b2-4059-8d5f-a06d13f5a84f" path="/var/lib/kubelet/pods/e0f445be-95b2-4059-8d5f-a06d13f5a84f/volumes"
Oct 30 03:19:19.184023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.183944    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-326068"]
Oct 30 03:19:19.190207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.190182    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:19.203610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.201337    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-326068"]
Oct 30 03:19:19.205674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.205652    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q6n5\""
Oct 30 03:19:19.301551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.301520    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdnw\" (UniqueName: \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\") pod \"test-curl-8800-326068\" (UID: \"c620cfcd-89f1-4469-8834-d09b4ba4c5f3\") " pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:19.319908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.319870    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" event={"ID":"bef7941a-6f8b-45ec-b2cb-5e2b892e0164","Type":"ContainerStarted","Data":"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"}
Oct 30 03:19:19.320416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.319926    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" event={"ID":"bef7941a-6f8b-45ec-b2cb-5e2b892e0164","Type":"ContainerStarted","Data":"d10be56b0261b9c245a4b34ccf7af11aebbc6c0fe34032b130a6832ef9a6a1e3"}
Oct 30 03:19:19.366888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.366688    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" podStartSLOduration=1.36667231 podStartE2EDuration="1.36667231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:19.361704084 +0000 UTC m=+2298.435874892" watchObservedRunningTime="2026-10-30 03:19:19.36667231 +0000 UTC m=+2298.440843158"
Oct 30 03:19:19.403602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.403560    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdnw\" (UniqueName: \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\") pod \"test-curl-8800-326068\" (UID: \"c620cfcd-89f1-4469-8834-d09b4ba4c5f3\") " pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:19.423748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.423720    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdnw\" (UniqueName: \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\") pod \"test-curl-8800-326068\" (UID: \"c620cfcd-89f1-4469-8834-d09b4ba4c5f3\") " pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:19.503259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.503156    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:19.510520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.510483    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:19.834105 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:19.834077    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc620cfcd_89f1_4469_8834_d09b4ba4c5f3.slice/crio-b32556035f29c5d787226afdc324e6901d911b6e10ca534d90fe4d33c1d06171 WatchSource:0}: Error finding container b32556035f29c5d787226afdc324e6901d911b6e10ca534d90fe4d33c1d06171: Status 404 returned error can't find the container with id b32556035f29c5d787226afdc324e6901d911b6e10ca534d90fe4d33c1d06171
Oct 30 03:19:19.834280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:19.834181    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-326068"]
Oct 30 03:19:20.331711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:20.331675    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-326068" event={"ID":"c620cfcd-89f1-4469-8834-d09b4ba4c5f3","Type":"ContainerStarted","Data":"2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472"}
Oct 30 03:19:20.331711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:20.331722    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-326068" event={"ID":"c620cfcd-89f1-4469-8834-d09b4ba4c5f3","Type":"ContainerStarted","Data":"b32556035f29c5d787226afdc324e6901d911b6e10ca534d90fe4d33c1d06171"}
Oct 30 03:19:20.374212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:20.374155    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-326068" podStartSLOduration=1.374139645 podStartE2EDuration="1.374139645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:20.368556349 +0000 UTC m=+2299.442727142" watchObservedRunningTime="2026-10-30 03:19:20.374139645 +0000 UTC m=+2299.448310437"
Oct 30 03:19:21.258552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.257498    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-326068"]
Oct 30 03:19:21.265332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.265030    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-326068"]
Oct 30 03:19:21.406149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.406101    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8800-e6ff78"]
Oct 30 03:19:21.406971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.406939    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" containerName="test-curl-8800-326068"
Oct 30 03:19:21.406971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.406968    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" containerName="test-curl-8800-326068"
Oct 30 03:19:21.407137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.407093    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" containerName="test-curl-8800-326068"
Oct 30 03:19:21.411115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.411097    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:21.431297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.431270    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-e6ff78"]
Oct 30 03:19:21.543003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.542914    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplq8\" (UniqueName: \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\") pod \"test-curl-8800-e6ff78\" (UID: \"eb4e4770-114a-420f-ab78-1a31495a3e62\") " pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:21.645018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.644978    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplq8\" (UniqueName: \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\") pod \"test-curl-8800-e6ff78\" (UID: \"eb4e4770-114a-420f-ab78-1a31495a3e62\") " pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:21.664942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.664915    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplq8\" (UniqueName: \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\") pod \"test-curl-8800-e6ff78\" (UID: \"eb4e4770-114a-420f-ab78-1a31495a3e62\") " pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:21.726119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:21.726090    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:22.031169 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:22.031138    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4e4770_114a_420f_ab78_1a31495a3e62.slice/crio-c129d98a53f28d0b6cfc3b9fa5af2ab062723f1f0968d73d2a1f3528050aa0a4 WatchSource:0}: Error finding container c129d98a53f28d0b6cfc3b9fa5af2ab062723f1f0968d73d2a1f3528050aa0a4: Status 404 returned error can't find the container with id c129d98a53f28d0b6cfc3b9fa5af2ab062723f1f0968d73d2a1f3528050aa0a4
Oct 30 03:19:22.032936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:22.032911    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8800-e6ff78"]
Oct 30 03:19:22.380945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:22.380887    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-326068" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" containerName="test-curl-8800-326068" containerID="cri-o://2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472" gracePeriod=2
Oct 30 03:19:22.381648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:22.381606    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-e6ff78" event={"ID":"eb4e4770-114a-420f-ab78-1a31495a3e62","Type":"ContainerStarted","Data":"c129d98a53f28d0b6cfc3b9fa5af2ab062723f1f0968d73d2a1f3528050aa0a4"}
Oct 30 03:19:22.408770 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:22.407166    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-326068\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:23.393469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:23.393431    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8800-e6ff78" event={"ID":"eb4e4770-114a-420f-ab78-1a31495a3e62","Type":"ContainerStarted","Data":"c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662"}
Oct 30 03:19:23.432149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:23.432096    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8800-e6ff78" podStartSLOduration=2.43208343 podStartE2EDuration="2.43208343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:23.431151383 +0000 UTC m=+2302.505322176" watchObservedRunningTime="2026-10-30 03:19:23.43208343 +0000 UTC m=+2302.506254222"
Oct 30 03:19:24.297829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.297794    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:19:24.298059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.298033    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy" containerID="cri-o://c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1" gracePeriod=30
Oct 30 03:19:24.376221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.376180    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8800-e6ff78"]
Oct 30 03:19:24.379413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.379378    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8800-e6ff78"]
Oct 30 03:19:24.908387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.908359    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:24.981818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.981728    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\" (UniqueName: \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\") pod \"c620cfcd-89f1-4469-8834-d09b4ba4c5f3\" (UID: \"c620cfcd-89f1-4469-8834-d09b4ba4c5f3\") "
Oct 30 03:19:24.983917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:24.983889    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw" pod "c620cfcd-89f1-4469-8834-d09b4ba4c5f3" (UID: "c620cfcd-89f1-4469-8834-d09b4ba4c5f3"). InnerVolumeSpecName "kube-api-access-hgdnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:25.069979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.069948    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" path="/var/lib/kubelet/pods/c620cfcd-89f1-4469-8834-d09b4ba4c5f3/volumes"
Oct 30 03:19:25.083777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.083688    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdnw\" (UniqueName: \"kubernetes.io/projected/c620cfcd-89f1-4469-8834-d09b4ba4c5f3-kube-api-access-hgdnw\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:25.611470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.611436    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:19:25.612296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.612280    4214 cpu_manager.go:414] "containerMap: removing container" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" containerName="test-curl-8800-e6ff78"
Oct 30 03:19:25.612349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.612298    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" containerName="test-curl-8800-e6ff78"
Oct 30 03:19:25.612480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.612470    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" containerName="test-curl-8800-e6ff78"
Oct 30 03:19:25.616763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.616742    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.639183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.639153    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:19:25.645536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.645501    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-086aee-openshift-default-dockercfg-s64bf\""
Oct 30 03:19:25.683654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.683618    4214 generic.go:356] "Generic (PLEG): container finished" podID="c620cfcd-89f1-4469-8834-d09b4ba4c5f3" containerID="2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472" exitCode=137
Oct 30 03:19:25.683846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.683708    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-326068"
Oct 30 03:19:25.683846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.683745    4214 scope.go:122] "RemoveContainer" containerID="2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472"
Oct 30 03:19:25.683974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.683845    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8800-e6ff78" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" containerName="test-curl-8800-e6ff78" containerID="cri-o://c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662" gracePeriod=2
Oct 30 03:19:25.690672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690648    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wb4\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690728    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690760    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690804    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690841    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690941 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690925    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.690987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.690971    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.691031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.691000    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.691031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.691022    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.697405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.697385    4214 scope.go:122] "RemoveContainer" containerID="2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472"
Oct 30 03:19:25.697707 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:25.697688    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472\": container with ID starting with 2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472 not found: ID does not exist" containerID="2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472"
Oct 30 03:19:25.697795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.697719    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472"} err="failed to get container status \"2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472\": rpc error: code = NotFound desc = could not find container \"2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472\": container with ID starting with 2c5a99cde1cc0590dedfc3206e052d9ba01966383480c7c993beff1a7c51c472 not found: ID does not exist"
Oct 30 03:19:25.792059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792024    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.792262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792105    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.792262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792215    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.792419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792321    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.792752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792673    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.792752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792742    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.793181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.792901    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wb4\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.793181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.793057    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.793387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.793193    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.794769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.794423    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.795711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.795107    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.795711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.795224    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.795711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.795366    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.795711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.795509    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.795711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.795670    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.844065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.843844    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.877661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.870614    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wb4\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.877661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.872554    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\") pod \"e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") " pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:25.938274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:25.937536    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:26.099250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.098997    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:26.394659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.394633    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:19:26.395802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.395767    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:19:26.395928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.395835    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:19:26.395928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.395864    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:19:26.697530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.697442    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerStarted","Data":"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"}
Oct 30 03:19:26.697530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.697498    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerStarted","Data":"6956f25ea87e9385d2c596f21ff43f116d17f799219881132e1ddde07cf8b885"}
Oct 30 03:19:26.744964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.744916    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podStartSLOduration=1.74490222 podStartE2EDuration="1.74490222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:26.740466392 +0000 UTC m=+2305.814637187" watchObservedRunningTime="2026-10-30 03:19:26.74490222 +0000 UTC m=+2305.819073012"
Oct 30 03:19:26.940265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:26.940222    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:27.913244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:27.913201    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:27.936755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:27.936727    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\" (UniqueName: \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\") pod \"eb4e4770-114a-420f-ab78-1a31495a3e62\" (UID: \"eb4e4770-114a-420f-ab78-1a31495a3e62\") "
Oct 30 03:19:27.939406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:27.939333    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8" pod "eb4e4770-114a-420f-ab78-1a31495a3e62" (UID: "eb4e4770-114a-420f-ab78-1a31495a3e62"). InnerVolumeSpecName "kube-api-access-mplq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:27.948367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:27.948340    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:28.038895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.038862    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplq8\" (UniqueName: \"kubernetes.io/projected/eb4e4770-114a-420f-ab78-1a31495a3e62-kube-api-access-mplq8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:28.493691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.493647    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:28.512071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.512046    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:28.720851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.720807    4214 generic.go:356] "Generic (PLEG): container finished" podID="eb4e4770-114a-420f-ab78-1a31495a3e62" containerID="c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662" exitCode=137
Oct 30 03:19:28.721053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.720860    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:28.721053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.720884    4214 scope.go:122] "RemoveContainer" containerID="c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662"
Oct 30 03:19:28.721461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.721430    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:28.722511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.722485    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:19:28.734063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.734047    4214 scope.go:122] "RemoveContainer" containerID="c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662"
Oct 30 03:19:28.734379 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:28.734355    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662\": container with ID starting with c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662 not found: ID does not exist" containerID="c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662"
Oct 30 03:19:28.734459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.734383    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662"} err="failed to get container status \"c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662\": rpc error: code = NotFound desc = could not find container \"c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662\": container with ID starting with c27590668d89734438b4f5647614aa40d37fe0ff8e853ba9310f41411150f662 not found: ID does not exist"
Oct 30 03:19:28.736251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:28.736217    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:19:28.821268 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:28.821199    4214 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8800-e6ff78\" is forbidden: User \"system:node:ip-10-0-136-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-171.ec2.internal' and this object" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" pod="opendatahub/test-curl-8800-e6ff78"
Oct 30 03:19:29.070649 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.070611    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4e4770-114a-420f-ab78-1a31495a3e62" path="/var/lib/kubelet/pods/eb4e4770-114a-420f-ab78-1a31495a3e62/volumes"
Oct 30 03:19:29.594061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.594037    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:19:29.671593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.671558    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.671816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.671608    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.671816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.671643    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672068    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672121    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672303    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672383    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672415    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\") pod \"42204c5f-32ff-4325-89a8-ce0d537827aa\" (UID: \"42204c5f-32ff-4325-89a8-ce0d537827aa\") "
Oct 30 03:19:29.672726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672693    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.673001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.672974    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.673271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673245    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.673438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673414    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:29.673592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673419    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.673592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673456    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.673592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673479    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.673592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.673494    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.689263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.675872    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "kube-api-access-85ld8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:29.689263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.676010    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.689263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.676296    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:19:29.689263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.676664    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token" pod "42204c5f-32ff-4325-89a8-ce0d537827aa" (UID: "42204c5f-32ff-4325-89a8-ce0d537827aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:29.736805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.736770    4214 generic.go:356] "Generic (PLEG): container finished" podID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerID="c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1" exitCode=0
Oct 30 03:19:29.736954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.736942    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"
Oct 30 03:19:29.737929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.737901    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerDied","Data":"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"}
Oct 30 03:19:29.738134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.738110    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf" event={"ID":"42204c5f-32ff-4325-89a8-ce0d537827aa","Type":"ContainerDied","Data":"9f2df498b4383ca9b7807f9de88863185092085342812a871ff43dbaf9791784"}
Oct 30 03:19:29.738243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.738156    4214 scope.go:122] "RemoveContainer" containerID="c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"
Oct 30 03:19:29.755475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.755454    4214 scope.go:122] "RemoveContainer" containerID="f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"
Oct 30 03:19:29.767716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.767694    4214 scope.go:122] "RemoveContainer" containerID="c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"
Oct 30 03:19:29.768023 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:29.767976    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1\": container with ID starting with c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1 not found: ID does not exist" containerID="c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"
Oct 30 03:19:29.768149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.768014    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1"} err="failed to get container status \"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1\": rpc error: code = NotFound desc = could not find container \"c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1\": container with ID starting with c3fa87d453ff2aab486d305d2ec1a5b458c6a19ba9882126a4ad40881e0d66b1 not found: ID does not exist"
Oct 30 03:19:29.768149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.768040    4214 scope.go:122] "RemoveContainer" containerID="f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"
Oct 30 03:19:29.768456 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:29.768423    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2\": container with ID starting with f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2 not found: ID does not exist" containerID="f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"
Oct 30 03:19:29.768567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.768462    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2"} err="failed to get container status \"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2\": rpc error: code = NotFound desc = could not find container \"f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2\": container with ID starting with f8219a00c0701152a944f3fa9a66cb297661fc3f6d05ae3b40e29b997e1c23c2 not found: ID does not exist"
Oct 30 03:19:29.775242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775204    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.775345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775276    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.775345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775294    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.775345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775307    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42204c5f-32ff-4325-89a8-ce0d537827aa-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.775345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775320    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42204c5f-32ff-4325-89a8-ce0d537827aa-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.775345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.775335    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ld8\" (UniqueName: \"kubernetes.io/projected/42204c5f-32ff-4325-89a8-ce0d537827aa-kube-api-access-85ld8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.797026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.797000    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:19:29.801376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:29.801358    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-kh7mf"]
Oct 30 03:19:30.225252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.214740    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:19:30.225252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.215003    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy" containerID="cri-o://72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66" gracePeriod=30
Oct 30 03:19:30.998964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.998924    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:19:30.999799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999781    4214 cpu_manager.go:414] "containerMap: removing container" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:30.999874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999800    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:30.999874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999829    4214 cpu_manager.go:414] "containerMap: removing container" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:30.999874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999836    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:30.999973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999936    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:30.999973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:30.999949    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" containerName="istio-proxy"
Oct 30 03:19:31.005502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.005478    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.022209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.022188    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:19:31.029298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.029016    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-086aee\""
Oct 30 03:19:31.029396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.029321    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-086aee-dockercfg-mm9qf\""
Oct 30 03:19:31.222624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.184191    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68t9p\" (UniqueName: \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.222624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.220446    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.328248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.325211    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68t9p\" (UniqueName: \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.328248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.325366    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.328248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.326606    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.354253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.346383    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68t9p\" (UniqueName: \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\") pod \"payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.399293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.398172    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42204c5f-32ff-4325-89a8-ce0d537827aa" path="/var/lib/kubelet/pods/42204c5f-32ff-4325-89a8-ce0d537827aa/volumes"
Oct 30 03:19:31.402012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.401979    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:19:31.408315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.408284    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:19:31.408429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.408375    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.528591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.528559    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j446f\" (UniqueName: \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.528851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.528829    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.623372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.623297    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:31.634879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.634841    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.635050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.635018    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j446f\" (UniqueName: \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.635715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.635692    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.656374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.656309    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j446f\" (UniqueName: \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\") pod \"payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.724573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.724540    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:31.949767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:31.949432    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:19:32.071035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.071006    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:19:32.072573 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:32.072549    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b411241_9efd_44f3_a97c_b778e2b4bf48.slice/crio-10e7a99b0f08e28742ef5710f0fd590c8f2f442f8db55b218720274f6b4d2522 WatchSource:0}: Error finding container 10e7a99b0f08e28742ef5710f0fd590c8f2f442f8db55b218720274f6b4d2522: Status 404 returned error can't find the container with id 10e7a99b0f08e28742ef5710f0fd590c8f2f442f8db55b218720274f6b4d2522
Oct 30 03:19:32.789825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.789777    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" event={"ID":"9b411241-9efd-44f3-a97c-b778e2b4bf48","Type":"ContainerStarted","Data":"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"}
Oct 30 03:19:32.790286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.789860    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:32.790286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.789886    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" event={"ID":"9b411241-9efd-44f3-a97c-b778e2b4bf48","Type":"ContainerStarted","Data":"10e7a99b0f08e28742ef5710f0fd590c8f2f442f8db55b218720274f6b4d2522"}
Oct 30 03:19:32.792265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.792213    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" event={"ID":"31dd1345-1ff9-463c-b202-bc3a1262cbc4","Type":"ContainerStarted","Data":"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"}
Oct 30 03:19:32.792396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.792290    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" event={"ID":"31dd1345-1ff9-463c-b202-bc3a1262cbc4","Type":"ContainerStarted","Data":"78eb0cc7c58ac94b3030cf3def70db50513c5dcb44a99b5980964cdb95ea5e1c"}
Oct 30 03:19:32.792396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.792321    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:32.833040 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.832990    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" podStartSLOduration=1.8329763049999999 podStartE2EDuration="1.832976305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:32.828996766 +0000 UTC m=+2311.903167559" watchObservedRunningTime="2026-10-30 03:19:32.832976305 +0000 UTC m=+2311.907147097"
Oct 30 03:19:32.868418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:32.868357    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" podStartSLOduration=2.868340533 podStartE2EDuration="2.868340533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:32.867436875 +0000 UTC m=+2311.941607680" watchObservedRunningTime="2026-10-30 03:19:32.868340533 +0000 UTC m=+2311.942511327"
Oct 30 03:19:33.595086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:33.595045    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:19:33.595348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:33.595313    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" podUID="9650431d-405f-43c3-a7ca-198df859657d" containerName="maas-api" containerID="cri-o://52afac48556bc83cfe80b6bbdfa473aee74af725def27b3481e3532c931aa601" gracePeriod=30
Oct 30 03:19:33.806905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:33.806871    4214 generic.go:356] "Generic (PLEG): container finished" podID="9650431d-405f-43c3-a7ca-198df859657d" containerID="52afac48556bc83cfe80b6bbdfa473aee74af725def27b3481e3532c931aa601" exitCode=0
Oct 30 03:19:33.807329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:33.806951    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" event={"ID":"9650431d-405f-43c3-a7ca-198df859657d","Type":"ContainerDied","Data":"52afac48556bc83cfe80b6bbdfa473aee74af725def27b3481e3532c931aa601"}
Oct 30 03:19:34.172752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.172721    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:19:34.246187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.245150    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\") pod \"9650431d-405f-43c3-a7ca-198df859657d\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") "
Oct 30 03:19:34.246187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.245494    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\") pod \"9650431d-405f-43c3-a7ca-198df859657d\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") "
Oct 30 03:19:34.246187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.245602    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\" (UniqueName: \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\") pod \"9650431d-405f-43c3-a7ca-198df859657d\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") "
Oct 30 03:19:34.246187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.245680    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\") pod \"9650431d-405f-43c3-a7ca-198df859657d\" (UID: \"9650431d-405f-43c3-a7ca-198df859657d\") "
Oct 30 03:19:34.247543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.247513    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca" pod "9650431d-405f-43c3-a7ca-198df859657d" (UID: "9650431d-405f-43c3-a7ca-198df859657d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:34.251184 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.251145    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs" pod "9650431d-405f-43c3-a7ca-198df859657d" (UID: "9650431d-405f-43c3-a7ca-198df859657d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:34.253136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.253098    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls" pod "9650431d-405f-43c3-a7ca-198df859657d" (UID: "9650431d-405f-43c3-a7ca-198df859657d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:34.274662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.274632    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv" pod "9650431d-405f-43c3-a7ca-198df859657d" (UID: "9650431d-405f-43c3-a7ca-198df859657d"). InnerVolumeSpecName "kube-api-access-mwscv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:34.349518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.349482    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.349518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.349515    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwscv\" (UniqueName: \"kubernetes.io/projected/9650431d-405f-43c3-a7ca-198df859657d-kube-api-access-mwscv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.349518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.349527    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9650431d-405f-43c3-a7ca-198df859657d-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.349852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.349536    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9650431d-405f-43c3-a7ca-198df859657d-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:34.858587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.858554    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"
Oct 30 03:19:34.858587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.858575    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc" event={"ID":"9650431d-405f-43c3-a7ca-198df859657d","Type":"ContainerDied","Data":"b554e274840861f3461a620973b817cfce347e0e07c0ccd3fce662d6680fb251"}
Oct 30 03:19:34.859164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.858648    4214 scope.go:122] "RemoveContainer" containerID="52afac48556bc83cfe80b6bbdfa473aee74af725def27b3481e3532c931aa601"
Oct 30 03:19:34.914521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.914454    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:19:34.927149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:34.927113    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f21058-5dd658dfb9-tctvc"]
Oct 30 03:19:35.070923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.070889    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9650431d-405f-43c3-a7ca-198df859657d" path="/var/lib/kubelet/pods/9650431d-405f-43c3-a7ca-198df859657d/volumes"
Oct 30 03:19:35.549176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.549150    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:19:35.672106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672067    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672195    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672310    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672379    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672532    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672587    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672636    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.672823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.672805    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\") pod \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\" (UID: \"06e8fffa-bf6d-404c-8da4-87f0e3d0a464\") "
Oct 30 03:19:35.673278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.673253    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:35.673376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.673319    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:35.673480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.673438    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:35.673542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.673500    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:35.673641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.673622    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:35.674119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.674097    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.674268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.674252    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.674375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.674363    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.674464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.674453    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.674539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.674531    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.676472 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.676396    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:35.676864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.676841    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:35.676950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.676877    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "kube-api-access-5t7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:35.677027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.677010    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo" pod "06e8fffa-bf6d-404c-8da4-87f0e3d0a464" (UID: "06e8fffa-bf6d-404c-8da4-87f0e3d0a464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:19:35.775965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.775887    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7gf\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-kube-api-access-5t7gf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.775965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.775916    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.775965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.775925    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.775965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.775933    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e8fffa-bf6d-404c-8da4-87f0e3d0a464-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.875277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.875020    4214 generic.go:356] "Generic (PLEG): container finished" podID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerID="72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66" exitCode=0
Oct 30 03:19:35.875277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.875056    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerDied","Data":"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"}
Oct 30 03:19:35.875277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.875094    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"
Oct 30 03:19:35.875277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.875130    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m" event={"ID":"06e8fffa-bf6d-404c-8da4-87f0e3d0a464","Type":"ContainerDied","Data":"a98cf744b50860a074f493f2d30897ace4936ee1ee7abada73ddbd8eb78efa69"}
Oct 30 03:19:35.875277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.875146    4214 scope.go:122] "RemoveContainer" containerID="72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"
Oct 30 03:19:35.891917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.891601    4214 scope.go:122] "RemoveContainer" containerID="43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"
Oct 30 03:19:35.902128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.902102    4214 scope.go:122] "RemoveContainer" containerID="72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"
Oct 30 03:19:35.902425 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:35.902405    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66\": container with ID starting with 72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66 not found: ID does not exist" containerID="72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"
Oct 30 03:19:35.902495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.902433    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66"} err="failed to get container status \"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66\": rpc error: code = NotFound desc = could not find container \"72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66\": container with ID starting with 72dc5686a1e1ed840443f8f935d15bb22c04c403232e73c49261eda8219cea66 not found: ID does not exist"
Oct 30 03:19:35.902495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.902448    4214 scope.go:122] "RemoveContainer" containerID="43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"
Oct 30 03:19:35.902661 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:35.902644    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d\": container with ID starting with 43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d not found: ID does not exist" containerID="43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"
Oct 30 03:19:35.902707 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.902665    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d"} err="failed to get container status \"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d\": rpc error: code = NotFound desc = could not find container \"43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d\": container with ID starting with 43bf69307fb612b8543dc20c3c28642b9b923e321a136b13dc5d75c7c34bd72d not found: ID does not exist"
Oct 30 03:19:35.945980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.945800    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:19:35.953135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:35.953090    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7f767-openshift-default-7d54d7f747-82h7m"]
Oct 30 03:19:36.097844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:36.097801    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:37.071325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.071282    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" path="/var/lib/kubelet/pods/06e8fffa-bf6d-404c-8da4-87f0e3d0a464/volumes"
Oct 30 03:19:37.711991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.711965    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:19:37.810616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.810585    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\" (UniqueName: \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\") pod \"76d15b39-6856-440f-b72e-61ab258ab2a7\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") "
Oct 30 03:19:37.810961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.810934    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\") pod \"76d15b39-6856-440f-b72e-61ab258ab2a7\" (UID: \"76d15b39-6856-440f-b72e-61ab258ab2a7\") "
Oct 30 03:19:37.811379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.811355    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume" pod "76d15b39-6856-440f-b72e-61ab258ab2a7" (UID: "76d15b39-6856-440f-b72e-61ab258ab2a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:37.811678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.811654    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76d15b39-6856-440f-b72e-61ab258ab2a7-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:37.812807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.812787    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz" pod "76d15b39-6856-440f-b72e-61ab258ab2a7" (UID: "76d15b39-6856-440f-b72e-61ab258ab2a7"). InnerVolumeSpecName "kube-api-access-6zqvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:37.896124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.896090    4214 generic.go:356] "Generic (PLEG): container finished" podID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerID="f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53" exitCode=0
Oct 30 03:19:37.896312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.896133    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"
Oct 30 03:19:37.896312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.896172    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" event={"ID":"76d15b39-6856-440f-b72e-61ab258ab2a7","Type":"ContainerDied","Data":"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"}
Oct 30 03:19:37.896312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.896258    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h" event={"ID":"76d15b39-6856-440f-b72e-61ab258ab2a7","Type":"ContainerDied","Data":"99885e465582745e9377e311acdb3e43d7992d021cdfed76ae83f6ee35512fe9"}
Oct 30 03:19:37.896312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.896286    4214 scope.go:122] "RemoveContainer" containerID="f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"
Oct 30 03:19:37.908028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.908013    4214 scope.go:122] "RemoveContainer" containerID="f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"
Oct 30 03:19:37.908312 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:37.908289    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53\": container with ID starting with f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53 not found: ID does not exist" containerID="f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"
Oct 30 03:19:37.908361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.908327    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53"} err="failed to get container status \"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53\": rpc error: code = NotFound desc = could not find container \"f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53\": container with ID starting with f23a79af0cdea5a60269df322299f14878cd9a047f8080a092ca3ef07cd35a53 not found: ID does not exist"
Oct 30 03:19:37.912654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.912633    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqvz\" (UniqueName: \"kubernetes.io/projected/76d15b39-6856-440f-b72e-61ab258ab2a7-kube-api-access-6zqvz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:37.956128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.956059    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:19:37.960509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:37.960485    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7f767-7c4dc78559-26s6h"]
Oct 30 03:19:38.702057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.701868    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:19:38.703015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.702989    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9650431d-405f-43c3-a7ca-198df859657d" containerName="maas-api"
Oct 30 03:19:38.703015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703012    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9650431d-405f-43c3-a7ca-198df859657d" containerName="maas-api"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703039    4214 cpu_manager.go:414] "containerMap: removing container" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703046    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703057    4214 cpu_manager.go:414] "containerMap: removing container" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703061    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703201    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:19:38.703221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703217    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9650431d-405f-43c3-a7ca-198df859657d" containerName="maas-api"
Oct 30 03:19:38.703547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.703244    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" containerName="payload-pre-processing"
Oct 30 03:19:38.708417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.708395    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.727628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.727599    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:19:38.734368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.734340    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"rate-test-model-360818-kserve-self-signed-certs\""
Oct 30 03:19:38.734500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.734408    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"default-dockercfg-jsdxd\""
Oct 30 03:19:38.736983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.736962    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"kube-root-ca.crt\""
Oct 30 03:19:38.737865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.737842    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"openshift-service-ca.crt\""
Oct 30 03:19:38.739406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.739384    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.739681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.739658    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.739895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.739875    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwpv\" (UniqueName: \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.740038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.740023    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.740206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.740189    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.740379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.740362    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842168    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwpv\" (UniqueName: \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842256    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842311    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842348    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842623    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.842714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.842695    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.843350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.843129    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.843350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.843162    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.845914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.845889    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.848470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.848449    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:38.862084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:38.862063    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwpv\" (UniqueName: \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\") pod \"rate-test-model-360818-kserve-564b476c45-5llx9\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") " pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:39.065334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.042628    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:39.112746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.112711    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d15b39-6856-440f-b72e-61ab258ab2a7" path="/var/lib/kubelet/pods/76d15b39-6856-440f-b72e-61ab258ab2a7/volumes"
Oct 30 03:19:39.172213 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:39.172155    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:19:39.424912 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:19:39.424878    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2141cb37_2374_4ba7_8fcb_58f9c0dcc8e0.slice/crio-796eff80952ad486a5298eb2153a68654a73887fd86f673d0051e3662b797adf WatchSource:0}: Error finding container 796eff80952ad486a5298eb2153a68654a73887fd86f673d0051e3662b797adf: Status 404 returned error can't find the container with id 796eff80952ad486a5298eb2153a68654a73887fd86f673d0051e3662b797adf
Oct 30 03:19:39.426077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.426051    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:19:39.914866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.914830    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" event={"ID":"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0","Type":"ContainerStarted","Data":"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"}
Oct 30 03:19:39.915320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.914875    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" event={"ID":"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0","Type":"ContainerStarted","Data":"796eff80952ad486a5298eb2153a68654a73887fd86f673d0051e3662b797adf"}
Oct 30 03:19:39.955873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:39.955820    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" podStartSLOduration=1.955806777 podStartE2EDuration="1.955806777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:39.955413815 +0000 UTC m=+2319.029584607" watchObservedRunningTime="2026-10-30 03:19:39.955806777 +0000 UTC m=+2319.029977603"
Oct 30 03:19:40.042929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:40.042896    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:41.189912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:41.189875    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:19:41.302604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:41.302572    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:19:41.302819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:41.302798    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy" containerID="cri-o://53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc" gracePeriod=30
Oct 30 03:19:43.860745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:43.860493    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:19:43.862599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:43.862574    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:19:46.190879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.190829    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" podUID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" containerName="payload-processing" containerID="cri-o://62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4" gracePeriod=25
Oct 30 03:19:46.525672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.525333    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:19:46.628308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.628280    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\" (UniqueName: \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\") pod \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") "
Oct 30 03:19:46.628431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.628389    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\") pod \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\" (UID: \"3f58dc2c-02c8-42a1-86ae-d4c513b30263\") "
Oct 30 03:19:46.628744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.628720    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume" pod "3f58dc2c-02c8-42a1-86ae-d4c513b30263" (UID: "3f58dc2c-02c8-42a1-86ae-d4c513b30263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:46.629139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.629116    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f58dc2c-02c8-42a1-86ae-d4c513b30263-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.630294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.630273    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc" pod "3f58dc2c-02c8-42a1-86ae-d4c513b30263" (UID: "3f58dc2c-02c8-42a1-86ae-d4c513b30263"). InnerVolumeSpecName "kube-api-access-b4ksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:46.649168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.649144    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:19:46.729676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729641    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.729853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729704    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.729853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729784    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.729853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729830    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729926    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.729970    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.730109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730025    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730080    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730136    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730157    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.730368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730211    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.730368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730298    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\") pod \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\" (UID: \"bd5ac6e4-1bad-4b53-9a91-fee6b1942195\") "
Oct 30 03:19:46.730592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730489    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.730784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730761    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:46.731007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.730985    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.731102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.731013    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.731102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.731029    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.731102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.731044    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.731102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.731057    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ksc\" (UniqueName: \"kubernetes.io/projected/3f58dc2c-02c8-42a1-86ae-d4c513b30263-kube-api-access-b4ksc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.731102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.731070    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.733022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.732948    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:46.733514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.733480    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:19:46.733514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.733487    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "kube-api-access-q4l54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:46.733938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.733914    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy" pod "bd5ac6e4-1bad-4b53-9a91-fee6b1942195" (UID: "bd5ac6e4-1bad-4b53-9a91-fee6b1942195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:46.833060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.833025    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.833060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.833055    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4l54\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-kube-api-access-q4l54\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.833060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.833066    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:46.833344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:46.833075    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5ac6e4-1bad-4b53-9a91-fee6b1942195-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:19:47.213491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.209033    4214 generic.go:356] "Generic (PLEG): container finished" podID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerID="53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc" exitCode=0
Oct 30 03:19:47.213491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.209276    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerDied","Data":"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"}
Oct 30 03:19:47.213491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.209333    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg" event={"ID":"bd5ac6e4-1bad-4b53-9a91-fee6b1942195","Type":"ContainerDied","Data":"9fe7c95f7a568fa281ef6ca2bc307404b228944b453ba0c1276dac5de1baed03"}
Oct 30 03:19:47.213491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.209367    4214 scope.go:122] "RemoveContainer" containerID="53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"
Oct 30 03:19:47.213491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.209553    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"
Oct 30 03:19:47.226716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.226689    4214 generic.go:356] "Generic (PLEG): container finished" podID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" containerID="62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4" exitCode=0
Oct 30 03:19:47.230395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.227169    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" event={"ID":"3f58dc2c-02c8-42a1-86ae-d4c513b30263","Type":"ContainerDied","Data":"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"}
Oct 30 03:19:47.234399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.231564    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"
Oct 30 03:19:47.241018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.239943    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k" event={"ID":"3f58dc2c-02c8-42a1-86ae-d4c513b30263","Type":"ContainerDied","Data":"963096ff1e09437ff9bdd3b64dea9209d591d1d4dbd1094b9ae0329c0ecc99bf"}
Oct 30 03:19:47.272422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.272390    4214 scope.go:122] "RemoveContainer" containerID="71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"
Oct 30 03:19:47.301612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.301587    4214 scope.go:122] "RemoveContainer" containerID="53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"
Oct 30 03:19:47.302184 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:47.302159    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc\": container with ID starting with 53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc not found: ID does not exist" containerID="53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"
Oct 30 03:19:47.302681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.302652    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc"} err="failed to get container status \"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc\": rpc error: code = NotFound desc = could not find container \"53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc\": container with ID starting with 53f9b14e1e7ca2567951479e496beb39e1ab90ecd19e3feef93b11585506aadc not found: ID does not exist"
Oct 30 03:19:47.302799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.302787    4214 scope.go:122] "RemoveContainer" containerID="71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"
Oct 30 03:19:47.303205 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:47.303183    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0\": container with ID starting with 71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0 not found: ID does not exist" containerID="71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"
Oct 30 03:19:47.303387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.303359    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0"} err="failed to get container status \"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0\": rpc error: code = NotFound desc = could not find container \"71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0\": container with ID starting with 71b9e59aa3f2e9e0a4f71ff7751a69008ea2dd5f20733b3746544c9ca0e43af0 not found: ID does not exist"
Oct 30 03:19:47.303495 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.303484    4214 scope.go:122] "RemoveContainer" containerID="62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"
Oct 30 03:19:47.324804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.324779    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:19:47.331663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.331637    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-100ffd-6454564456-7jm7k"]
Oct 30 03:19:47.331864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.331841    4214 scope.go:122] "RemoveContainer" containerID="62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"
Oct 30 03:19:47.332562 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:19:47.332293    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4\": container with ID starting with 62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4 not found: ID does not exist" containerID="62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"
Oct 30 03:19:47.332562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.332335    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4"} err="failed to get container status \"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4\": rpc error: code = NotFound desc = could not find container \"62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4\": container with ID starting with 62b0652497991cbbd7ffff45f3eb97fa01be5c3171d237e5ad3159d19640dda4 not found: ID does not exist"
Oct 30 03:19:47.402965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.385277    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:19:47.439099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:47.439066    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f21058-openshift-default-57fbbd47dc-gfjpg"]
Oct 30 03:19:49.042892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:49.042852    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:49.071060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:49.071029    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" path="/var/lib/kubelet/pods/3f58dc2c-02c8-42a1-86ae-d4c513b30263/volumes"
Oct 30 03:19:49.071660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:49.071642    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" path="/var/lib/kubelet/pods/bd5ac6e4-1bad-4b53-9a91-fee6b1942195/volumes"
Oct 30 03:19:49.072124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:49.072099    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:49.296858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:49.296833    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:19:56.466254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:56.466207    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:19:56.499891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:56.499860    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:19:59.166436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:59.166393    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:19:59.166863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:19:59.166482    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:20:00.485625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.485593    4214 generic.go:356] "Generic (PLEG): container finished" podID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerID="777084db5f9b9a18e15d87f6266b43aa27c334b666e9482e6766b0ddaa1e9097" exitCode=0
Oct 30 03:20:00.486089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.485662    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerDied","Data":"777084db5f9b9a18e15d87f6266b43aa27c334b666e9482e6766b0ddaa1e9097"}
Oct 30 03:20:00.486168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.486098    4214 scope.go:122] "RemoveContainer" containerID="777084db5f9b9a18e15d87f6266b43aa27c334b666e9482e6766b0ddaa1e9097"
Oct 30 03:20:00.487942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.487915    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:00.488016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.487989    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:00.488250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:00.488023    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:01.468257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:01.467626    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" podUID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" containerName="payload-processing" containerID="cri-o://ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e" gracePeriod=25
Oct 30 03:20:01.501185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:01.501147    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" podUID="c1730013-6ae5-4703-843a-063439e5c4ec" containerName="payload-pre-processing" containerID="cri-o://b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a" gracePeriod=25
Oct 30 03:20:01.504675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:01.504644    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerStarted","Data":"659733fafe122146975dd4672d2da057a4ad0ad772f0ea450edd08c8a0adfde0"}
Oct 30 03:20:01.921156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:01.921122    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:20:02.060280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.055933    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\" (UniqueName: \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\") pod \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") "
Oct 30 03:20:02.060280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.056030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\") pod \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\" (UID: \"31a56217-364b-4a4c-8f8c-a3dadeb4eddd\") "
Oct 30 03:20:02.060280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.057112    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume" pod "31a56217-364b-4a4c-8f8c-a3dadeb4eddd" (UID: "31a56217-364b-4a4c-8f8c-a3dadeb4eddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:02.083487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.068472    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:20:02.114251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.113189    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm" pod "31a56217-364b-4a4c-8f8c-a3dadeb4eddd" (UID: "31a56217-364b-4a4c-8f8c-a3dadeb4eddd"). InnerVolumeSpecName "kube-api-access-9dwgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:02.162028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.161985    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\") pod \"c1730013-6ae5-4703-843a-063439e5c4ec\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") "
Oct 30 03:20:02.162241 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.162199    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\" (UniqueName: \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\") pod \"c1730013-6ae5-4703-843a-063439e5c4ec\" (UID: \"c1730013-6ae5-4703-843a-063439e5c4ec\") "
Oct 30 03:20:02.169298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.162834    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwgm\" (UniqueName: \"kubernetes.io/projected/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-kube-api-access-9dwgm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:02.169298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.162865    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a56217-364b-4a4c-8f8c-a3dadeb4eddd-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:02.169298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.166478    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume" pod "c1730013-6ae5-4703-843a-063439e5c4ec" (UID: "c1730013-6ae5-4703-843a-063439e5c4ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:02.169298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.169072    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z" pod "c1730013-6ae5-4703-843a-063439e5c4ec" (UID: "c1730013-6ae5-4703-843a-063439e5c4ec"). InnerVolumeSpecName "kube-api-access-x826z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:02.223643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.223610    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:20:02.223943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.223914    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 03:20:02.224017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.223989    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 03:20:02.267725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.267574    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1730013-6ae5-4703-843a-063439e5c4ec-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:02.267725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.267615    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x826z\" (UniqueName: \"kubernetes.io/projected/c1730013-6ae5-4703-843a-063439e5c4ec-kube-api-access-x826z\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:02.514176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.514146    4214 generic.go:356] "Generic (PLEG): container finished" podID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" containerID="ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e" exitCode=0
Oct 30 03:20:02.514655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.514193    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"
Oct 30 03:20:02.514655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.514239    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" event={"ID":"31a56217-364b-4a4c-8f8c-a3dadeb4eddd","Type":"ContainerDied","Data":"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"}
Oct 30 03:20:02.514655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.514287    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr" event={"ID":"31a56217-364b-4a4c-8f8c-a3dadeb4eddd","Type":"ContainerDied","Data":"fad4a30d530d76e137f1a06ee5857b6a5811daeb7dfc1a616b9db6c1dacee32f"}
Oct 30 03:20:02.514655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.514301    4214 scope.go:122] "RemoveContainer" containerID="ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"
Oct 30 03:20:02.516160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.516135    4214 generic.go:356] "Generic (PLEG): container finished" podID="c1730013-6ae5-4703-843a-063439e5c4ec" containerID="b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a" exitCode=0
Oct 30 03:20:02.516289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.516197    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"
Oct 30 03:20:02.516289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.516205    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" event={"ID":"c1730013-6ae5-4703-843a-063439e5c4ec","Type":"ContainerDied","Data":"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"}
Oct 30 03:20:02.516289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.516259    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v" event={"ID":"c1730013-6ae5-4703-843a-063439e5c4ec","Type":"ContainerDied","Data":"7c17e5fdcec289f60a590874d670e62e875e4af202830632299c11483ce56c0f"}
Oct 30 03:20:02.528609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.528586    4214 scope.go:122] "RemoveContainer" containerID="ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"
Oct 30 03:20:02.528907 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:02.528887    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e\": container with ID starting with ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e not found: ID does not exist" containerID="ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"
Oct 30 03:20:02.528985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.528915    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e"} err="failed to get container status \"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e\": rpc error: code = NotFound desc = could not find container \"ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e\": container with ID starting with ba8b3e8addf9616a20edeb13ffd382de8497f7b95072e553c36530be9fd92d4e not found: ID does not exist"
Oct 30 03:20:02.528985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.528930    4214 scope.go:122] "RemoveContainer" containerID="b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"
Oct 30 03:20:02.558417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.554966    4214 scope.go:122] "RemoveContainer" containerID="b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"
Oct 30 03:20:02.558417 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:02.555348    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a\": container with ID starting with b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a not found: ID does not exist" containerID="b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"
Oct 30 03:20:02.558417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.555382    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a"} err="failed to get container status \"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a\": rpc error: code = NotFound desc = could not find container \"b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a\": container with ID starting with b8227971a65f9df9dbee7948a73e24d7ff576996a0ca4411a7975658abd2804a not found: ID does not exist"
Oct 30 03:20:02.577040 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.577009    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:20:02.587785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.587745    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-554566bb99-vfv6v"]
Oct 30 03:20:02.624457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.624427    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:20:02.628696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:02.628671    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-f7846c9b-jbxrr"]
Oct 30 03:20:03.079801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:03.079768    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" path="/var/lib/kubelet/pods/31a56217-364b-4a4c-8f8c-a3dadeb4eddd/volumes"
Oct 30 03:20:03.080389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:03.080357    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1730013-6ae5-4703-843a-063439e5c4ec" path="/var/lib/kubelet/pods/c1730013-6ae5-4703-843a-063439e5c4ec/volumes"
Oct 30 03:20:03.223696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:03.223655    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 03:20:03.223896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:03.223727    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 03:20:04.224436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:04.224405    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 03:20:04.224839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:04.224454    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 03:20:05.224051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:05.224011    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 03:20:05.224250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:05.224068    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 03:20:06.224124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:06.224084    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 03:20:06.224654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:06.224163    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 03:20:07.131598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.131563    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:20:07.132381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132364    4214 cpu_manager.go:414] "containerMap: removing container" podUID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" containerName="payload-processing"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132382    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" containerName="payload-processing"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132399    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c1730013-6ae5-4703-843a-063439e5c4ec" containerName="payload-pre-processing"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132406    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1730013-6ae5-4703-843a-063439e5c4ec" containerName="payload-pre-processing"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132421    4214 cpu_manager.go:414] "containerMap: removing container" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132426    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132436    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132440    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132449    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132454    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132471    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" containerName="payload-processing"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132476    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" containerName="payload-processing"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132574    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a56217-364b-4a4c-8f8c-a3dadeb4eddd" containerName="payload-processing"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132584    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1730013-6ae5-4703-843a-063439e5c4ec" containerName="payload-pre-processing"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132592    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132600    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5ac6e4-1bad-4b53-9a91-fee6b1942195" containerName="istio-proxy"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132609    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e8fffa-bf6d-404c-8da4-87f0e3d0a464" containerName="istio-proxy"
Oct 30 03:20:07.132749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.132615    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f58dc2c-02c8-42a1-86ae-d4c513b30263" containerName="payload-processing"
Oct 30 03:20:07.136873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.136855    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.160905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.160881    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-086aee-serving-cert\""
Oct 30 03:20:07.177893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.177855    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:20:07.228751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.228718    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:20:07.229273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.229247    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:20:07.230186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.230164    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:20:07.249897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.249862    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.250075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.250008    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.250276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.250246    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.250345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.250297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrwpf\" (UniqueName: \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.352360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.352322    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.352360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.352365    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrwpf\" (UniqueName: \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.352654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.352514    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.352725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.352652    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.353162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.353127    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.355151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.355126    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.355332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.355263    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.380308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.380275    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrwpf\" (UniqueName: \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\") pod \"maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.454466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.454375    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:07.869757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:07.869728    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:20:07.870453 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:07.870422    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d6bf98_c3dd_49f1_8c9f_97cee84c25f1.slice/crio-7d68ed5d396b33c520fb5e6000091e5b6696dd3495e9a89958b61e7dd1bb4f2b WatchSource:0}: Error finding container 7d68ed5d396b33c520fb5e6000091e5b6696dd3495e9a89958b61e7dd1bb4f2b: Status 404 returned error can't find the container with id 7d68ed5d396b33c520fb5e6000091e5b6696dd3495e9a89958b61e7dd1bb4f2b
Oct 30 03:20:08.646296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:08.646224    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" event={"ID":"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1","Type":"ContainerStarted","Data":"9ff889cb818635641a2889d1ff0b253a86bf08496c56af5b2f80edbae7c3e70a"}
Oct 30 03:20:08.646748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:08.646336    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:08.646748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:08.646358    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" event={"ID":"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1","Type":"ContainerStarted","Data":"7d68ed5d396b33c520fb5e6000091e5b6696dd3495e9a89958b61e7dd1bb4f2b"}
Oct 30 03:20:08.690123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:08.690062    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" podStartSLOduration=1.336055045 podStartE2EDuration="1.690038315s" totalImagesPullingTime="353.98327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:08.688386996 +0000 UTC m=+2347.762557789" watchObservedRunningTime="2026-10-30 03:20:08.690038315 +0000 UTC m=+2347.764209110"
Oct 30 03:20:09.402254 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:09.400398    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:20:14.680950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:14.680919    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:20:16.714251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:16.714145    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:20:16.714807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:16.714409    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" containerID="cri-o://2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7" gracePeriod=30
Oct 30 03:20:19.892104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:19.892062    4214 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:20:19.892512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:19.892130    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:21.503383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:21.503348    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:20:21.503907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:21.503576    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerName="maas-api" containerID="cri-o://8bda2b1f7e8667ec017c7bd7d35e99d9c9651f7c7adf71b1bc2df2c397af9dd1" gracePeriod=30
Oct 30 03:20:21.825254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:21.824987    4214 generic.go:356] "Generic (PLEG): container finished" podID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerID="8bda2b1f7e8667ec017c7bd7d35e99d9c9651f7c7adf71b1bc2df2c397af9dd1" exitCode=0
Oct 30 03:20:21.825433 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:21.825280    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" event={"ID":"b0fd346f-62cd-4bba-8f36-52b078f8cd19","Type":"ContainerDied","Data":"8bda2b1f7e8667ec017c7bd7d35e99d9c9651f7c7adf71b1bc2df2c397af9dd1"}
Oct 30 03:20:22.137274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.131077    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.203221    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\") pod \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") "
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.204207    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\" (UniqueName: \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\") pod \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") "
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.204271    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\") pod \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") "
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.204347    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") pod \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\" (UID: \"b0fd346f-62cd-4bba-8f36-52b078f8cd19\") "
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.205785    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca" pod "b0fd346f-62cd-4bba-8f36-52b078f8cd19" (UID: "b0fd346f-62cd-4bba-8f36-52b078f8cd19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:22.208248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.206955    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd346f-62cd-4bba-8f36-52b078f8cd19-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.211487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.209751    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls" pod "b0fd346f-62cd-4bba-8f36-52b078f8cd19" (UID: "b0fd346f-62cd-4bba-8f36-52b078f8cd19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:22.211487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.211141    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps" pod "b0fd346f-62cd-4bba-8f36-52b078f8cd19" (UID: "b0fd346f-62cd-4bba-8f36-52b078f8cd19"). InnerVolumeSpecName "kube-api-access-g8bps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.213593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.212749    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs" pod "b0fd346f-62cd-4bba-8f36-52b078f8cd19" (UID: "b0fd346f-62cd-4bba-8f36-52b078f8cd19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:22.303059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.302992    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:20:22.309072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.309038    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bps\" (UniqueName: \"kubernetes.io/projected/b0fd346f-62cd-4bba-8f36-52b078f8cd19-kube-api-access-g8bps\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.309210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.309075    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.309210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.309090    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd346f-62cd-4bba-8f36-52b078f8cd19-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.410429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410395    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410472    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410513    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410539    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410669    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410724    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.410832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410726    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.410832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410812    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.411138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410937    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\") pod \"a0fbe39d-99c0-49c7-a57b-629e536706e3\" (UID: \"a0fbe39d-99c0-49c7-a57b-629e536706e3\") "
Oct 30 03:20:22.411138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.410958    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.411303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.411282    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:22.411686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.411657    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.411686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.411687    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0fbe39d-99c0-49c7-a57b-629e536706e3-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.411835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.411704    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.412112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.412078    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.412327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.412293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.413632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.413570    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "kube-api-access-hdpwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.413887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.413859    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.413887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.413868    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:22.414051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.413956    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy" pod "a0fbe39d-99c0-49c7-a57b-629e536706e3" (UID: "a0fbe39d-99c0-49c7-a57b-629e536706e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.513180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513149    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.513180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513177    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.513180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513187    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdpwq\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-kube-api-access-hdpwq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.513795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513195    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.513795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513204    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0fbe39d-99c0-49c7-a57b-629e536706e3-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.513795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.513213    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0fbe39d-99c0-49c7-a57b-629e536706e3-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.842943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.842895    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" event={"ID":"b0fd346f-62cd-4bba-8f36-52b078f8cd19","Type":"ContainerDied","Data":"f2500cb19d75b9bce7a67e1209fd30c1b61e815c9bf76b7ca526b9aec7642769"}
Oct 30 03:20:22.843166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.843001    4214 scope.go:122] "RemoveContainer" containerID="8bda2b1f7e8667ec017c7bd7d35e99d9c9651f7c7adf71b1bc2df2c397af9dd1"
Oct 30 03:20:22.843166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.842907    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"
Oct 30 03:20:22.851632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.850092    4214 generic.go:356] "Generic (PLEG): container finished" podID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerID="2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7" exitCode=0
Oct 30 03:20:22.851632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.850310    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerDied","Data":"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"}
Oct 30 03:20:22.851632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.850362    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw" event={"ID":"a0fbe39d-99c0-49c7-a57b-629e536706e3","Type":"ContainerDied","Data":"bfdac6d3c54ea531a0ffa6e535811481da88bf23bf45bb54250118e9f8cd02c7"}
Oct 30 03:20:22.851632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.850449    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"
Oct 30 03:20:22.863757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.863733    4214 scope.go:122] "RemoveContainer" containerID="2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"
Oct 30 03:20:22.879635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.879609    4214 scope.go:122] "RemoveContainer" containerID="42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"
Oct 30 03:20:22.893085 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.893064    4214 scope.go:122] "RemoveContainer" containerID="2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"
Oct 30 03:20:22.893487 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:22.893462    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7\": container with ID starting with 2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7 not found: ID does not exist" containerID="2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"
Oct 30 03:20:22.893560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.893498    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7"} err="failed to get container status \"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7\": rpc error: code = NotFound desc = could not find container \"2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7\": container with ID starting with 2d9145a9acd15b47f8542c3efcb171275fa570a40bc9b00bf57fbc02346fc3c7 not found: ID does not exist"
Oct 30 03:20:22.893560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.893517    4214 scope.go:122] "RemoveContainer" containerID="42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"
Oct 30 03:20:22.893848 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:22.893819    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768\": container with ID starting with 42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768 not found: ID does not exist" containerID="42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"
Oct 30 03:20:22.893917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.893852    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768"} err="failed to get container status \"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768\": rpc error: code = NotFound desc = could not find container \"42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768\": container with ID starting with 42b6a2c3e6af6858e5d61580942b003eea83b80b03e27f09bd04b61691ca6768 not found: ID does not exist"
Oct 30 03:20:22.925434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.925399    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:20:22.932991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.932958    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6"]
Oct 30 03:20:22.986121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.986082    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:20:22.990404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:22.990372    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-lngvw"]
Oct 30 03:20:23.073368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.073331    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" path="/var/lib/kubelet/pods/a0fbe39d-99c0-49c7-a57b-629e536706e3/volumes"
Oct 30 03:20:23.074018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.073996    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" path="/var/lib/kubelet/pods/b0fd346f-62cd-4bba-8f36-52b078f8cd19/volumes"
Oct 30 03:20:23.618858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.618820    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:20:23.620037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620010    4214 cpu_manager.go:414] "containerMap: removing container" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:23.620037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620037    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:23.620273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620050    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerName="maas-api"
Oct 30 03:20:23.620273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620059    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerName="maas-api"
Oct 30 03:20:23.620273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620148    4214 cpu_manager.go:414] "containerMap: removing container" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:23.620273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620156    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:23.620493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620364    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:23.620493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.620383    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerName="maas-api"
Oct 30 03:20:23.625854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.625834    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.656122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.656071    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:20:23.663200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.663167    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-146bb7-openshift-default-dockercfg-qwvlq\""
Oct 30 03:20:23.750997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.742047    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.754358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.754327    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.756392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.756122    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk86\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.756924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.756901    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.757371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.757350    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.757706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.757652    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.758176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.758156    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.758455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.758436    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.762886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.762860    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.869028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.867867    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.869409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.869382    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.869593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.869577    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.869755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.869739    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.869996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.869981    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.871764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.870662    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.871918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.871564    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.872015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.871657    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.872111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.871729    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.872621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.872604    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk86\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.873011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.872994    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.876325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.876302    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.883571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.877109    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.883985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.876126    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.934272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.924474    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.934272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.927643    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.934272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.929185    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.945272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.945037    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk86\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\") pod \"e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") " pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:23.955456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:23.955418    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:24.385888 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:24.385858    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf32909d_505e_4f7a_8afa_8ebfb3af15de.slice/crio-1e6fe937af6682ef9b7682e8e91c34f568a71567e709f86f9bdb0e8ebed9a38c WatchSource:0}: Error finding container 1e6fe937af6682ef9b7682e8e91c34f568a71567e709f86f9bdb0e8ebed9a38c: Status 404 returned error can't find the container with id 1e6fe937af6682ef9b7682e8e91c34f568a71567e709f86f9bdb0e8ebed9a38c
Oct 30 03:20:24.386579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.386553    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:20:24.389096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.389064    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:24.389196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.389132    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:24.389196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.389161    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:24.898759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.898717    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerStarted","Data":"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"}
Oct 30 03:20:24.899203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.898788    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerStarted","Data":"1e6fe937af6682ef9b7682e8e91c34f568a71567e709f86f9bdb0e8ebed9a38c"}
Oct 30 03:20:24.958254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.957852    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:24.961221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.961165    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podStartSLOduration=1.961150988 podStartE2EDuration="1.961150988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:24.956882198 +0000 UTC m=+2364.031052990" watchObservedRunningTime="2026-10-30 03:20:24.961150988 +0000 UTC m=+2364.035321780"
Oct 30 03:20:24.980242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:24.980194    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2305f4-556f75ff44-9hzd6" podUID="b0fd346f-62cd-4bba-8f36-52b078f8cd19" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.232:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:20:25.983620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:25.983583    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:26.963359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:26.963323    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:26.964429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:26.964408    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:20:28.753768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.747953    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:20:28.753768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.749369    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fbe39d-99c0-49c7-a57b-629e536706e3" containerName="istio-proxy"
Oct 30 03:20:28.807477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.794435    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.857077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.827784    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.857077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.827840    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjmx\" (UniqueName: \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.857077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.828269    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07e5c2\""
Oct 30 03:20:28.857077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.850617    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07e5c2-dockercfg-lph5t\""
Oct 30 03:20:28.919334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.794330    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:20:28.919598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.919584    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:20:28.928673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.928650    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:28.929796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.929484    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.929796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.929537    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjmx\" (UniqueName: \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.932280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.930371    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:28.932829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.932805    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:20:28.991241 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:28.991185    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjmx\" (UniqueName: \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\") pod \"payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:29.032500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.031894    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.032500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.032124    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9kf\" (UniqueName: \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.133767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.133704    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9kf\" (UniqueName: \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.133942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.133909    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.134793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.134761    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.155651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.155625    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9kf\" (UniqueName: \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\") pod \"payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.226479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.226450    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:29.251048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.251017    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:29.546448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.546412    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:20:29.551518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.551489    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.568759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.568581    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-146bb7\""
Oct 30 03:20:29.568759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.568698    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:20:29.568759 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:29.568726    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe390f2c_87b9_4f51_b0d6_59f185ac2c5f.slice/crio-18966aea9564194fe09ce800eb0c5195beaf193b3ffb4b0b782fd2059d37accf WatchSource:0}: Error finding container 18966aea9564194fe09ce800eb0c5195beaf193b3ffb4b0b782fd2059d37accf: Status 404 returned error can't find the container with id 18966aea9564194fe09ce800eb0c5195beaf193b3ffb4b0b782fd2059d37accf
Oct 30 03:20:29.569625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.569608    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-146bb7-dockercfg-m9jxf\""
Oct 30 03:20:29.582014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.581988    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:20:29.664251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.654255    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8sn\" (UniqueName: \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.664251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.654498    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.771278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.755792    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.771278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.755942    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8sn\" (UniqueName: \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.771278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.757208    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.771278 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:29.766275    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1991f52c_7032_403a_8ab0_6e02949aa979.slice/crio-666857447ba4c7c9b436acb65f3cade054ef8b41835a8c0d1f7fa71f29a3811c WatchSource:0}: Error finding container 666857447ba4c7c9b436acb65f3cade054ef8b41835a8c0d1f7fa71f29a3811c: Status 404 returned error can't find the container with id 666857447ba4c7c9b436acb65f3cade054ef8b41835a8c0d1f7fa71f29a3811c
Oct 30 03:20:29.807381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.806147    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:20:29.820122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.820017    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8sn\" (UniqueName: \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.868647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.868611    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:29.883315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.883283    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:20:29.900849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.900827    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:20:29.901067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.901025    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:29.963248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.961061    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:29.963248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:29.961199    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzltt\" (UniqueName: \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.009543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.009506    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" event={"ID":"1991f52c-7032-403a-8ab0-6e02949aa979","Type":"ContainerStarted","Data":"666857447ba4c7c9b436acb65f3cade054ef8b41835a8c0d1f7fa71f29a3811c"}
Oct 30 03:20:30.011280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.011213    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" event={"ID":"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f","Type":"ContainerStarted","Data":"18966aea9564194fe09ce800eb0c5195beaf193b3ffb4b0b782fd2059d37accf"}
Oct 30 03:20:30.062442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.062409    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzltt\" (UniqueName: \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.062638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.062620    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.063366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.063341    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.094918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.094833    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzltt\" (UniqueName: \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\") pod \"payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.217428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.217394    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:30.229697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.229670    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:20:30.232123 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:30.232091    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdda8fe_f076_48dd_bd2b_9fabc523dd1b.slice/crio-b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d WatchSource:0}: Error finding container b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d: Status 404 returned error can't find the container with id b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d
Oct 30 03:20:30.587665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:30.586549    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:20:30.590846 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:30.590817    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef02104c_7af8_4fab_9cb1_7c09afeb1f77.slice/crio-0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d WatchSource:0}: Error finding container 0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d: Status 404 returned error can't find the container with id 0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d
Oct 30 03:20:31.035998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.035912    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" event={"ID":"acdda8fe-f076-48dd-bd2b-9fabc523dd1b","Type":"ContainerStarted","Data":"99525bf0eadfab529543ebc9899d701cceca00c5e008d11a1af817091a38918a"}
Oct 30 03:20:31.035998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.035974    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" event={"ID":"acdda8fe-f076-48dd-bd2b-9fabc523dd1b","Type":"ContainerStarted","Data":"b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d"}
Oct 30 03:20:31.036565 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.036038    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:31.037932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.037903    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" event={"ID":"ef02104c-7af8-4fab-9cb1-7c09afeb1f77","Type":"ContainerStarted","Data":"3bdf5a054ec8ed29915ff667962f44e8d9945429a50917f2ed81035bf239b998"}
Oct 30 03:20:31.038072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.037954    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" event={"ID":"ef02104c-7af8-4fab-9cb1-7c09afeb1f77","Type":"ContainerStarted","Data":"0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d"}
Oct 30 03:20:31.038072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.037997    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:31.119781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.119718    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" podStartSLOduration=2.119701713 podStartE2EDuration="2.119701713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:31.112979632 +0000 UTC m=+2370.187150428" watchObservedRunningTime="2026-10-30 03:20:31.119701713 +0000 UTC m=+2370.193872507"
Oct 30 03:20:31.163530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.163453    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" podStartSLOduration=2.163432972 podStartE2EDuration="2.163432972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:31.157442584 +0000 UTC m=+2370.231613379" watchObservedRunningTime="2026-10-30 03:20:31.163432972 +0000 UTC m=+2370.237603769"
Oct 30 03:20:31.837314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.837238    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:20:31.837554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:31.837484    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy" containerID="cri-o://2b581d3a6e70e7bb65bc667f1d152f559a1aae2e6a0532dc0ec310033e7369bd" gracePeriod=30
Oct 30 03:20:33.214459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.213030    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" event={"ID":"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f","Type":"ContainerStarted","Data":"df8f8ceedfcffe7ec2a6856b00bfe8f5b9b0ff3833ded492814e5f38bbd4bfbb"}
Oct 30 03:20:33.214459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.214390    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:33.224894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.224759    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" event={"ID":"1991f52c-7032-403a-8ab0-6e02949aa979","Type":"ContainerStarted","Data":"62671c77e0ac7fea83876eb8aa8c70f84db039de44365e02eb51cbf215828e14"}
Oct 30 03:20:33.224894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.224832    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:33.264728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.264679    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:20:33.301479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.293323    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" podStartSLOduration=2.084575976 podStartE2EDuration="5.293302865s" totalImagesPullingTime="3.208726889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:33.288804728 +0000 UTC m=+2372.362975521" watchObservedRunningTime="2026-10-30 03:20:33.293302865 +0000 UTC m=+2372.367473659"
Oct 30 03:20:33.324501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.324471    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:20:33.324879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.324808    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.348338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.346665    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-d823d1-openshift-default-dockercfg-bhzs8\""
Oct 30 03:20:33.374750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.374699    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" podStartSLOduration=2.387327267 podStartE2EDuration="5.37468416s" totalImagesPullingTime="2.987356893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:33.369804142 +0000 UTC m=+2372.443974935" watchObservedRunningTime="2026-10-30 03:20:33.37468416 +0000 UTC m=+2372.448854954"
Oct 30 03:20:33.504303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.497296    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.504303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.497417    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.504303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.497459    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsscz\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.504303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.497488    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.520252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.517575    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.520252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.517667    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.520252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.517724    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.520252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.517762    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.520252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.517861    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621446    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621526    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621571    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621601    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621674    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621789    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621880    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621916    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsscz\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621940    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.621943    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.622130    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.622331    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.622426    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.624944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.622921    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.625939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.625183    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.626892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.626861    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.643447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.643422    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsscz\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.643693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.643671    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:33.651111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:33.651084    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:34.027928 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:34.027893    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf592998c_cb32_4292_a4ac_86eca80e305e.slice/crio-b0bba3942f1d8f46d5fe6ac3134338c4a897fc9e4f87b0e124fce36bcaa008bf WatchSource:0}: Error finding container b0bba3942f1d8f46d5fe6ac3134338c4a897fc9e4f87b0e124fce36bcaa008bf: Status 404 returned error can't find the container with id b0bba3942f1d8f46d5fe6ac3134338c4a897fc9e4f87b0e124fce36bcaa008bf
Oct 30 03:20:34.031904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.031875    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:20:34.032018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.031952    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:34.032082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.032019    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:34.032082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.032048    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:34.242246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.242185    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerStarted","Data":"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"}
Oct 30 03:20:34.242740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.242261    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerStarted","Data":"b0bba3942f1d8f46d5fe6ac3134338c4a897fc9e4f87b0e124fce36bcaa008bf"}
Oct 30 03:20:34.294064 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.294000    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podStartSLOduration=1.293978612 podStartE2EDuration="1.293978612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:34.291498229 +0000 UTC m=+2373.365669025" watchObservedRunningTime="2026-10-30 03:20:34.293978612 +0000 UTC m=+2373.368149406"
Oct 30 03:20:34.651989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:34.651903    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:35.659607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:35.659572    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:36.262382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:36.262350    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:36.263404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:36.263380    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:20:37.287519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.284108    4214 generic.go:356] "Generic (PLEG): container finished" podID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerID="2b581d3a6e70e7bb65bc667f1d152f559a1aae2e6a0532dc0ec310033e7369bd" exitCode=0
Oct 30 03:20:37.287519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.284196    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerDied","Data":"2b581d3a6e70e7bb65bc667f1d152f559a1aae2e6a0532dc0ec310033e7369bd"}
Oct 30 03:20:37.287519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.284314    4214 scope.go:122] "RemoveContainer" containerID="c770c796ace441054a6dbd444f2baee2ef64a7e359a1a157d0aedec409b8211c"
Oct 30 03:20:37.352328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.352301    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:20:37.419512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419470    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.419722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419538    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.419722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419589    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.419857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419844    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.419923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419884    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.419923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419917    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.420033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.419955    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.420033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.420001    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.420142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.420030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\") pod \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\" (UID: \"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7\") "
Oct 30 03:20:37.420142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.420105    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.420404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.420383    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.420927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.420902    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.422125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.422107    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.422286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.421489    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.422432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.421565    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:37.422563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.421804    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.424316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.424293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "kube-api-access-gbmtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:37.424441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.424319    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:37.426197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.426173    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:37.427388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.427367    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy" pod "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" (UID: "73dc1e3d-6a59-45ec-adaa-e951c3f3ced7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.523919    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmtt\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-kube-api-access-gbmtt\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.523951    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.523965    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.523976    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.523988    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.524001    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.524369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:37.524013    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:38.372593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.371734    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"
Oct 30 03:20:38.377646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.377617    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk" event={"ID":"73dc1e3d-6a59-45ec-adaa-e951c3f3ced7","Type":"ContainerDied","Data":"e68d0d43008508c3d228028a7aab0d8aa42e58d13b9034d2e61a51d54e07e799"}
Oct 30 03:20:38.377869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.377857    4214 scope.go:122] "RemoveContainer" containerID="2b581d3a6e70e7bb65bc667f1d152f559a1aae2e6a0532dc0ec310033e7369bd"
Oct 30 03:20:38.447063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.447028    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:20:38.463149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.463108    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-t84tk"]
Oct 30 03:20:38.811497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.811459    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:20:38.812253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812219    4214 cpu_manager.go:414] "containerMap: removing container" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.812320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812255    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.812320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812299    4214 cpu_manager.go:414] "containerMap: removing container" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.812320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812304    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.812435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812420    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.812475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.812434    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" containerName="istio-proxy"
Oct 30 03:20:38.817558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.817541    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:38.847386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.847361    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-d823d1-dockercfg-7q2q4\""
Oct 30 03:20:38.849910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.849886    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-d823d1\""
Oct 30 03:20:38.862493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.853483    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:20:39.026254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.993660    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rs6\" (UniqueName: \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.026254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:38.993968    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.095989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.095898    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.096415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.096390    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rs6\" (UniqueName: \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.098501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.098479    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.109338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.109255    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73dc1e3d-6a59-45ec-adaa-e951c3f3ced7" path="/var/lib/kubelet/pods/73dc1e3d-6a59-45ec-adaa-e951c3f3ced7/volumes"
Oct 30 03:20:39.110046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.110023    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:20:39.122200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.121703    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:20:39.122685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.122664    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.181847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.181809    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rs6\" (UniqueName: \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\") pod \"payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.190349 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:39.190297    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:20:39.198211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.198175    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28c49\" (UniqueName: \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.198407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.198364    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.299928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.299899    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28c49\" (UniqueName: \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.300103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.300048    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.300753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.300733    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.339833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.339797    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28c49\" (UniqueName: \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\") pod \"payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.432103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.432025    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:39.503523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.503494    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:39.904089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:39.899490    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:20:40.042319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.035605    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:20:40.406395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.405365    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" event={"ID":"9c57d49a-895e-4077-8c68-4b7637713d87","Type":"ContainerStarted","Data":"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"}
Oct 30 03:20:40.406395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.405433    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" event={"ID":"9c57d49a-895e-4077-8c68-4b7637713d87","Type":"ContainerStarted","Data":"7504f166b421c8670d1c5a688347a41aea2529c144a6a5fbe2171491ac935b0f"}
Oct 30 03:20:40.406656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.406438    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:40.408935 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.408911    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" event={"ID":"e32f8695-0896-40d7-9893-ac55d5d161b0","Type":"ContainerStarted","Data":"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"}
Oct 30 03:20:40.409052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.408950    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" event={"ID":"e32f8695-0896-40d7-9893-ac55d5d161b0","Type":"ContainerStarted","Data":"d0c5f36e37107f69b215ffae57803da0e2174165def7f3479a725defd12b8227"}
Oct 30 03:20:40.409262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.409213    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:40.503265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:40.502392    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" podStartSLOduration=2.502372461 podStartE2EDuration="2.502372461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:40.448872906 +0000 UTC m=+2379.523043701" watchObservedRunningTime="2026-10-30 03:20:40.502372461 +0000 UTC m=+2379.576543256"
Oct 30 03:20:42.169718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:42.168449    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:20:42.173494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:42.173462    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:20:42.214156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:42.214104    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" podStartSLOduration=4.214088589 podStartE2EDuration="4.214088589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:40.505899762 +0000 UTC m=+2379.580070556" watchObservedRunningTime="2026-10-30 03:20:42.214088589 +0000 UTC m=+2381.288259381"
Oct 30 03:20:44.259307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:44.256371    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:20:45.261263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:45.260374    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:20:50.131531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.131493    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:20:50.139476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.139449    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.158199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.158174    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-734274-openshift-default-dockercfg-dfz6s\""
Oct 30 03:20:50.173403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.173371    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:20:50.322375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.322331    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.322898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.322867    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvlb\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.323258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.323183    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.323453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.323426    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.323737 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.323719    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.323982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.323961    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.324178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.324159    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.324471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.324443    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.324661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.324643    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.426397    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.426483    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.426522    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.426606    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.426631    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.427516    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.427701    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.427809    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvlb\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.427857    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.427887    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.428306    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.428547    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.429091    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.430349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.430096    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.439253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.435885    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.476166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.476089    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvlb\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.480459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.480424    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.482387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.482362    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\") pod \"e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") " pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.762109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.762027    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:50.771089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.771058    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:20:50.814198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:50.814156    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:20:51.167251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.166961    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:20:51.171524 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:20:51.171483    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d38247_1126_46aa_9bc0_d15ac86cb2d1.slice/crio-3ed198ae99e008421af20020f2fa205ced4b2512907e5995c7e9354f16ccb481 WatchSource:0}: Error finding container 3ed198ae99e008421af20020f2fa205ced4b2512907e5995c7e9354f16ccb481: Status 404 returned error can't find the container with id 3ed198ae99e008421af20020f2fa205ced4b2512907e5995c7e9354f16ccb481
Oct 30 03:20:51.175296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.175259    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:51.175393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.175349    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:51.175449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.175392    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:51.436719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.436636    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:20:51.705027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.704934    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerStarted","Data":"e0fd857a00ec941296244fb4e4886bcaeb3445c75bc9cebd48cc6a58aebb757e"}
Oct 30 03:20:51.705298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.705272    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerStarted","Data":"3ed198ae99e008421af20020f2fa205ced4b2512907e5995c7e9354f16ccb481"}
Oct 30 03:20:51.756112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.756047    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podStartSLOduration=1.756027774 podStartE2EDuration="1.756027774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:51.75477521 +0000 UTC m=+2390.828946004" watchObservedRunningTime="2026-10-30 03:20:51.756027774 +0000 UTC m=+2390.830198570"
Oct 30 03:20:51.763245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:51.763204    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:52.465750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.465715    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:20:52.715151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.715119    4214 generic.go:356] "Generic (PLEG): container finished" podID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerID="c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655" exitCode=0
Oct 30 03:20:52.715363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.715189    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerDied","Data":"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"}
Oct 30 03:20:52.715811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.715749    4214 scope.go:122] "RemoveContainer" containerID="c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"
Oct 30 03:20:52.717624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.717596    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:52.717693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.717663    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:52.717743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.717699    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:20:52.771274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:52.770455    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:53.726128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.726093    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerStarted","Data":"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"}
Oct 30 03:20:53.726600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.726496    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:53.727337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.727318    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:20:53.967250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.942575    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:20:53.967250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.945833    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:53.967250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:53.945887    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:54.941597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:54.941562    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:54.942184 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:54.942152    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:55.771556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:55.771514    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" podUID="9b411241-9efd-44f3-a97c-b778e2b4bf48" containerName="payload-processing" containerID="cri-o://2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617" gracePeriod=25
Oct 30 03:20:55.815022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:55.814960    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" podUID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" containerName="payload-pre-processing" containerID="cri-o://afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b" gracePeriod=25
Oct 30 03:20:55.960313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:55.959302    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:55.960313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:55.959380    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:56.315695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.315668    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:20:56.354255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.350510    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:20:56.395252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.393933    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\") pod \"9b411241-9efd-44f3-a97c-b778e2b4bf48\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") "
Oct 30 03:20:56.395252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.394099    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\" (UniqueName: \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\") pod \"9b411241-9efd-44f3-a97c-b778e2b4bf48\" (UID: \"9b411241-9efd-44f3-a97c-b778e2b4bf48\") "
Oct 30 03:20:56.395252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.394620    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume" pod "9b411241-9efd-44f3-a97c-b778e2b4bf48" (UID: "9b411241-9efd-44f3-a97c-b778e2b4bf48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:56.395252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.395033    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b411241-9efd-44f3-a97c-b778e2b4bf48-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.397072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.397035    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f" pod "9b411241-9efd-44f3-a97c-b778e2b4bf48" (UID: "9b411241-9efd-44f3-a97c-b778e2b4bf48"). InnerVolumeSpecName "kube-api-access-j446f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:56.496067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.495757    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\" (UniqueName: \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\") pod \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") "
Oct 30 03:20:56.496067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.495838    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\") pod \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\" (UID: \"31dd1345-1ff9-463c-b202-bc3a1262cbc4\") "
Oct 30 03:20:56.496764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.496740    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j446f\" (UniqueName: \"kubernetes.io/projected/9b411241-9efd-44f3-a97c-b778e2b4bf48-kube-api-access-j446f\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.497020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.496987    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume" pod "31dd1345-1ff9-463c-b202-bc3a1262cbc4" (UID: "31dd1345-1ff9-463c-b202-bc3a1262cbc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:56.500290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.500254    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p" pod "31dd1345-1ff9-463c-b202-bc3a1262cbc4" (UID: "31dd1345-1ff9-463c-b202-bc3a1262cbc4"). InnerVolumeSpecName "kube-api-access-68t9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:56.598449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.598407    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68t9p\" (UniqueName: \"kubernetes.io/projected/31dd1345-1ff9-463c-b202-bc3a1262cbc4-kube-api-access-68t9p\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.598449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.598441    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd1345-1ff9-463c-b202-bc3a1262cbc4-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.811127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.811097    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:20:56.813734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.813707    4214 generic.go:356] "Generic (PLEG): container finished" podID="9b411241-9efd-44f3-a97c-b778e2b4bf48" containerID="2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617" exitCode=0
Oct 30 03:20:56.813922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.813903    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" event={"ID":"9b411241-9efd-44f3-a97c-b778e2b4bf48","Type":"ContainerDied","Data":"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"}
Oct 30 03:20:56.814039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.813955    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8" event={"ID":"9b411241-9efd-44f3-a97c-b778e2b4bf48","Type":"ContainerDied","Data":"10e7a99b0f08e28742ef5710f0fd590c8f2f442f8db55b218720274f6b4d2522"}
Oct 30 03:20:56.814039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.813989    4214 scope.go:122] "RemoveContainer" containerID="2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"
Oct 30 03:20:56.814201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.814150    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"
Oct 30 03:20:56.843742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.830538    4214 generic.go:356] "Generic (PLEG): container finished" podID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" containerID="afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b" exitCode=0
Oct 30 03:20:56.843742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.830686    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" event={"ID":"31dd1345-1ff9-463c-b202-bc3a1262cbc4","Type":"ContainerDied","Data":"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"}
Oct 30 03:20:56.843742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.830730    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb" event={"ID":"31dd1345-1ff9-463c-b202-bc3a1262cbc4","Type":"ContainerDied","Data":"78eb0cc7c58ac94b3030cf3def70db50513c5dcb44a99b5980964cdb95ea5e1c"}
Oct 30 03:20:56.843742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.830803    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"
Oct 30 03:20:56.896628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.896604    4214 scope.go:122] "RemoveContainer" containerID="2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"
Oct 30 03:20:56.898781 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:56.898744    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617\": container with ID starting with 2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617 not found: ID does not exist" containerID="2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"
Oct 30 03:20:56.898890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.898794    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617"} err="failed to get container status \"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617\": rpc error: code = NotFound desc = could not find container \"2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617\": container with ID starting with 2a96dd8b0e5c486847260aff5d4f548175b23f0b871903fd94b1aadcdfa4e617 not found: ID does not exist"
Oct 30 03:20:56.898890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.898818    4214 scope.go:122] "RemoveContainer" containerID="afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"
Oct 30 03:20:56.919773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.919747    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:20:56.927915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.927896    4214 scope.go:122] "RemoveContainer" containerID="afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"
Oct 30 03:20:56.928281 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:20:56.928252    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b\": container with ID starting with afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b not found: ID does not exist" containerID="afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"
Oct 30 03:20:56.928431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.928404    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b"} err="failed to get container status \"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b\": rpc error: code = NotFound desc = could not find container \"afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b\": container with ID starting with afc4c40231c6d77f3cd8c67fa3cc7f7837af2334e5e79d00a41f9b470191356b not found: ID does not exist"
Oct 30 03:20:56.931278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.931251    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-79f65bf7f8-cbnz8"]
Oct 30 03:20:56.966252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.952648    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:56.966252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:56.952714    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:57.050489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:57.050457    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:20:57.362357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:57.362319    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b411241-9efd-44f3-a97c-b778e2b4bf48" path="/var/lib/kubelet/pods/9b411241-9efd-44f3-a97c-b778e2b4bf48/volumes"
Oct 30 03:20:57.363140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:57.363101    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-57bbf65759-dzpzb"]
Oct 30 03:20:57.947501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:57.947464    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:57.947783 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:57.947753    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:58.721751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:58.721704    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:20:58.957253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:58.954416    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 03:20:58.957253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:58.954472    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 03:20:59.075698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:59.075661    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" path="/var/lib/kubelet/pods/31dd1345-1ff9-463c-b202-bc3a1262cbc4/volumes"
Oct 30 03:20:59.947484 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:59.947392    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:20:59.949102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:20:59.949075    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:21:07.611497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:07.611454    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:21:07.652541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:07.652500    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:21:09.169679 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:09.169636    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:21:11.976148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:11.976096    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" podUID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" containerName="main" containerID="cri-o://f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970" gracePeriod=45
Oct 30 03:21:12.291378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.291350    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:21:12.388632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.388590    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.388799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.388658    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.388860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.388835    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.388944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.388890    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\" (UniqueName: \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.389012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.388961    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.389080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.389015    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\") pod \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\" (UID: \"1730cd1a-43ff-46bd-ad97-847d0497ce8a\") "
Oct 30 03:21:12.389561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.389533    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:12.389774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.389732    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:12.389905 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.389871    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.389992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.389941    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:12.392141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.392104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:12.392141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.392129    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "kube-api-access-4cwp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:12.392341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.392180    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs" pod "1730cd1a-43ff-46bd-ad97-847d0497ce8a" (UID: "1730cd1a-43ff-46bd-ad97-847d0497ce8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:12.500353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.493517    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.500353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.493558    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.500353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.493571    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.500353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.493590    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1730cd1a-43ff-46bd-ad97-847d0497ce8a-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.500353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.493604    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwp2\" (UniqueName: \"kubernetes.io/projected/1730cd1a-43ff-46bd-ad97-847d0497ce8a-kube-api-access-4cwp2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.612792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.612750    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing" containerID="cri-o://07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80" gracePeriod=25
Oct 30 03:21:12.653655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:12.653600    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing" containerID="cri-o://c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee" gracePeriod=25
Oct 30 03:21:13.025124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.025008    4214 generic.go:356] "Generic (PLEG): container finished" podID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" containerID="f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970" exitCode=0
Oct 30 03:21:13.027106 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.025813    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"
Oct 30 03:21:13.028690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.025836    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" event={"ID":"1730cd1a-43ff-46bd-ad97-847d0497ce8a","Type":"ContainerDied","Data":"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"}
Oct 30 03:21:13.028690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.028215    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz" event={"ID":"1730cd1a-43ff-46bd-ad97-847d0497ce8a","Type":"ContainerDied","Data":"ec7fdd82bd2b3bf128f1708ebea8d25eebae59d3e19cbc0b4c0ad759948d1937"}
Oct 30 03:21:13.028690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.028259    4214 scope.go:122] "RemoveContainer" containerID="f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"
Oct 30 03:21:13.084876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.084850    4214 scope.go:122] "RemoveContainer" containerID="f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"
Oct 30 03:21:13.085835 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:13.085792    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970\": container with ID starting with f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970 not found: ID does not exist" containerID="f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"
Oct 30 03:21:13.086014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.085988    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970"} err="failed to get container status \"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970\": rpc error: code = NotFound desc = could not find container \"f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970\": container with ID starting with f0dafe42fef89b698ca09045da2d0a1dbf6cd31beb5a1ac60cc4bca533b55970 not found: ID does not exist"
Oct 30 03:21:13.119166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.119134    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:21:13.130806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:13.130778    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4cf8f5/test-model-4cf8f5-kserve-b77d66555-bqcvz"]
Oct 30 03:21:15.070638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:15.070605    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" path="/var/lib/kubelet/pods/1730cd1a-43ff-46bd-ad97-847d0497ce8a/volumes"
Oct 30 03:21:16.139767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.139726    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:21:16.140934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.140911    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" containerName="main"
Oct 30 03:21:16.141051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.140936    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" containerName="main"
Oct 30 03:21:16.141051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.140997    4214 cpu_manager.go:414] "containerMap: removing container" podUID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" containerName="payload-pre-processing"
Oct 30 03:21:16.141051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141006    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" containerName="payload-pre-processing"
Oct 30 03:21:16.141265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141077    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9b411241-9efd-44f3-a97c-b778e2b4bf48" containerName="payload-processing"
Oct 30 03:21:16.141265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141084    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b411241-9efd-44f3-a97c-b778e2b4bf48" containerName="payload-processing"
Oct 30 03:21:16.141265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141256    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1730cd1a-43ff-46bd-ad97-847d0497ce8a" containerName="main"
Oct 30 03:21:16.141432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141276    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dd1345-1ff9-463c-b202-bc3a1262cbc4" containerName="payload-pre-processing"
Oct 30 03:21:16.141432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.141296    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b411241-9efd-44f3-a97c-b778e2b4bf48" containerName="payload-processing"
Oct 30 03:21:16.147328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.147304    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.168050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.168017    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:21:16.168553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.168525    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-d823d1-serving-cert\""
Oct 30 03:21:16.255543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.255498    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.255728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.255631    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2z8\" (UniqueName: \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.255728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.255709    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.255868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.255852    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.361259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.360349    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.361259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.360489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.361259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.360615    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2z8\" (UniqueName: \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.361259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.360741    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.379315 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:16.369388    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-serving-cert: secret "maas-api-e2e-shared-a-w0-d823d1-serving-cert" not found
Oct 30 03:21:16.379315 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:16.369495    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls podName:559ed535-5e2e-4e43-9a1c-0b38dd3f2843 nodeName:}" failed. No retries permitted until 2026-10-30 03:21:16.869471089 +0000 UTC m=+2415.943641876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls") pod "maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" (UID: "559ed535-5e2e-4e43-9a1c-0b38dd3f2843") : secret "maas-api-e2e-shared-a-w0-d823d1-serving-cert" not found
Oct 30 03:21:16.379315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.370514    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.383252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.380614    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.395083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.394997    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2z8\" (UniqueName: \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:16.972252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:16.971054    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:17.005016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:17.004976    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:17.076016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:17.075985    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:17.466653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:17.466487    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:21:18.084906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:18.084866    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" event={"ID":"559ed535-5e2e-4e43-9a1c-0b38dd3f2843","Type":"ContainerStarted","Data":"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"}
Oct 30 03:21:18.085120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:18.084928    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" event={"ID":"559ed535-5e2e-4e43-9a1c-0b38dd3f2843","Type":"ContainerStarted","Data":"2aac41b92e6277a317ef753ab24e2cb524f446120a787e905ef33c4ad6494205"}
Oct 30 03:21:18.085120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:18.084962    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:18.982506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:18.982466    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:21:18.982911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:18.982565    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:21:24.099537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:24.099503    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:21:24.149384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:24.149308    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" podStartSLOduration=7.670291955 podStartE2EDuration="8.149289714s" totalImagesPullingTime="478.997759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:18.138774309 +0000 UTC m=+2417.212945108" watchObservedRunningTime="2026-10-30 03:21:24.149289714 +0000 UTC m=+2423.223460506"
Oct 30 03:21:28.983110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:28.983073    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:21:28.983718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:28.983058    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:21:33.090292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.090261    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:21:33.117763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.117740    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:21:33.142390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.142342    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\") pod \"436526a6-cd16-415c-a6b8-168ed98f8d1e\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") "
Oct 30 03:21:33.142598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.142425    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\" (UniqueName: \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\") pod \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") "
Oct 30 03:21:33.142598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.142486    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\") pod \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\" (UID: \"b60e8ccd-e20e-4d9b-9ebb-4614586b0562\") "
Oct 30 03:21:33.142726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.142640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\" (UniqueName: \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\") pod \"436526a6-cd16-415c-a6b8-168ed98f8d1e\" (UID: \"436526a6-cd16-415c-a6b8-168ed98f8d1e\") "
Oct 30 03:21:33.142807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.142775    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume" pod "436526a6-cd16-415c-a6b8-168ed98f8d1e" (UID: "436526a6-cd16-415c-a6b8-168ed98f8d1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:33.143496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.143468    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436526a6-cd16-415c-a6b8-168ed98f8d1e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:33.144761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.144715    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume" pod "b60e8ccd-e20e-4d9b-9ebb-4614586b0562" (UID: "b60e8ccd-e20e-4d9b-9ebb-4614586b0562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:33.146354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.146332    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts" pod "b60e8ccd-e20e-4d9b-9ebb-4614586b0562" (UID: "b60e8ccd-e20e-4d9b-9ebb-4614586b0562"). InnerVolumeSpecName "kube-api-access-kzwts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:33.147386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.147361    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn" pod "436526a6-cd16-415c-a6b8-168ed98f8d1e" (UID: "436526a6-cd16-415c-a6b8-168ed98f8d1e"). InnerVolumeSpecName "kube-api-access-6nkmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:33.245535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.245455    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzwts\" (UniqueName: \"kubernetes.io/projected/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-kube-api-access-kzwts\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:33.245535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.245485    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60e8ccd-e20e-4d9b-9ebb-4614586b0562-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:33.245535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.245497    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkmn\" (UniqueName: \"kubernetes.io/projected/436526a6-cd16-415c-a6b8-168ed98f8d1e-kube-api-access-6nkmn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:33.461009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.460971    4214 generic.go:356] "Generic (PLEG): container finished" podID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerID="07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80" exitCode=0
Oct 30 03:21:33.461195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.461041    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"
Oct 30 03:21:33.461195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.461061    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" event={"ID":"b60e8ccd-e20e-4d9b-9ebb-4614586b0562","Type":"ContainerDied","Data":"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"}
Oct 30 03:21:33.461195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.461127    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx" event={"ID":"b60e8ccd-e20e-4d9b-9ebb-4614586b0562","Type":"ContainerDied","Data":"deb2df0ba994130c245a8f018e94b3c2ba9817e01aa52245e988a9e59fb294f3"}
Oct 30 03:21:33.461195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.461159    4214 scope.go:122] "RemoveContainer" containerID="07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"
Oct 30 03:21:33.463193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.463167    4214 generic.go:356] "Generic (PLEG): container finished" podID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerID="c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee" exitCode=0
Oct 30 03:21:33.463369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.463213    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" event={"ID":"436526a6-cd16-415c-a6b8-168ed98f8d1e","Type":"ContainerDied","Data":"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"}
Oct 30 03:21:33.463369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.463289    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb" event={"ID":"436526a6-cd16-415c-a6b8-168ed98f8d1e","Type":"ContainerDied","Data":"fe341d8affcf7934a24280664b46d14d7d0c560e30b36b68ff44357c76b639cf"}
Oct 30 03:21:33.463369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.463360    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"
Oct 30 03:21:33.476037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.475933    4214 scope.go:122] "RemoveContainer" containerID="07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"
Oct 30 03:21:33.476377 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:33.476350    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80\": container with ID starting with 07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80 not found: ID does not exist" containerID="07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"
Oct 30 03:21:33.476621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.476586    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80"} err="failed to get container status \"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80\": rpc error: code = NotFound desc = could not find container \"07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80\": container with ID starting with 07c4076dea9e572c44b8df80184c57ef2c7533edaa280eea3e91a1b8ea6aba80 not found: ID does not exist"
Oct 30 03:21:33.476765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.476741    4214 scope.go:122] "RemoveContainer" containerID="c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"
Oct 30 03:21:33.488934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.488907    4214 scope.go:122] "RemoveContainer" containerID="c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"
Oct 30 03:21:33.489304 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:33.489256    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee\": container with ID starting with c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee not found: ID does not exist" containerID="c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"
Oct 30 03:21:33.489386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.489295    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee"} err="failed to get container status \"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee\": rpc error: code = NotFound desc = could not find container \"c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee\": container with ID starting with c8cf79915457942917d7c7af3a4867464a44d62f7a07b3f7249a689a018509ee not found: ID does not exist"
Oct 30 03:21:33.535496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.535404    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:21:33.551955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.551918    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0b40d-6c7b76b864-4lrtx"]
Oct 30 03:21:33.582048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.582006    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:21:33.587021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:33.586983    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0b40d-858d44967-gdzsb"]
Oct 30 03:21:35.074471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:35.074425    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" path="/var/lib/kubelet/pods/436526a6-cd16-415c-a6b8-168ed98f8d1e/volumes"
Oct 30 03:21:35.075198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:35.075175    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" path="/var/lib/kubelet/pods/b60e8ccd-e20e-4d9b-9ebb-4614586b0562/volumes"
Oct 30 03:21:37.506213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.506182    4214 generic.go:356] "Generic (PLEG): container finished" podID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerID="84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f" exitCode=0
Oct 30 03:21:37.506785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.506255    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerDied","Data":"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"}
Oct 30 03:21:37.506785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.506688    4214 scope.go:122] "RemoveContainer" containerID="84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"
Oct 30 03:21:37.508522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.508490    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:37.508646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.508577    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:37.508646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:37.508627    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:38.516209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:38.516176    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerStarted","Data":"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"}
Oct 30 03:21:38.958535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:38.958493    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:21:38.958923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:38.958897    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:38.959157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:38.959130    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:39.276714 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:39.276611    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:21:39.958637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:39.958598    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:39.959421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:39.959386    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:40.965799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:40.965762    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:40.966458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:40.966421    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:41.963254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:41.962617    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:41.963254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:41.962691    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:41.966450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:41.966420    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:21:42.178266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.153187    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:21:42.202591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.200451    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerName="maas-api" containerID="cri-o://a9a57e02dfe65636d8aac81f22b4ac5cd288958f50c8096a18496b61447b7311" gracePeriod=30
Oct 30 03:21:42.909774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.909740    4214 generic.go:356] "Generic (PLEG): container finished" podID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerID="a9a57e02dfe65636d8aac81f22b4ac5cd288958f50c8096a18496b61447b7311" exitCode=0
Oct 30 03:21:42.909971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.909792    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" event={"ID":"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec","Type":"ContainerDied","Data":"a9a57e02dfe65636d8aac81f22b4ac5cd288958f50c8096a18496b61447b7311"}
Oct 30 03:21:43.000263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.999523    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:43.000263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:42.999596    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:43.089834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.089806    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:21:43.198589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.198488    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\") pod \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") "
Oct 30 03:21:43.198790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.198600    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\") pod \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") "
Oct 30 03:21:43.198790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.198625    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\") pod \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") "
Oct 30 03:21:43.198790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.198706    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\") pod \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\" (UID: \"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec\") "
Oct 30 03:21:43.198990 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.198935    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca" pod "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" (UID: "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:43.199996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.199958    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:43.201156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.201110    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs" pod "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" (UID: "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:43.203370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.203345    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls" pod "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" (UID: "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:43.203462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.203379    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8" pod "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" (UID: "c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec"). InnerVolumeSpecName "kube-api-access-5cbv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:43.301869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.301833    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:43.301869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.301872    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-kube-api-access-5cbv8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:43.302079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.301888    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:43.923143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.923115    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"
Oct 30 03:21:43.923358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.923140    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" event={"ID":"c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec","Type":"ContainerDied","Data":"9facad2b393595261e46878ac59e9f4133452a09d50952f5dd191edaaa3dfa27"}
Oct 30 03:21:43.923358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.923205    4214 scope.go:122] "RemoveContainer" containerID="a9a57e02dfe65636d8aac81f22b4ac5cd288958f50c8096a18496b61447b7311"
Oct 30 03:21:43.960017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.959173    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Oct 30 03:21:43.960017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.959258    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Oct 30 03:21:43.983178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.983139    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:21:43.988456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:43.988424    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p"]
Oct 30 03:21:44.963391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:44.963360    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:21:44.964649 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:44.964627    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:21:45.105438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:45.105399    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" path="/var/lib/kubelet/pods/c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec/volumes"
Oct 30 03:21:45.700504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:45.700466    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b0b40d-6946cdd6d5-mtk9p" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.238:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:21:49.161657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.161618    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:21:49.162458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162440    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing"
Oct 30 03:21:49.162458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162457    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing"
Oct 30 03:21:49.162623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162491    4214 cpu_manager.go:414] "containerMap: removing container" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing"
Oct 30 03:21:49.162623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162505    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing"
Oct 30 03:21:49.162623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162533    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerName="maas-api"
Oct 30 03:21:49.162623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162539    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerName="maas-api"
Oct 30 03:21:49.162843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162710    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="436526a6-cd16-415c-a6b8-168ed98f8d1e" containerName="payload-pre-processing"
Oct 30 03:21:49.162843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162726    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d09a3c-2ec5-422d-93ad-cf6c3aeee3ec" containerName="maas-api"
Oct 30 03:21:49.162843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.162739    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60e8ccd-e20e-4d9b-9ebb-4614586b0562" containerName="payload-processing"
Oct 30 03:21:49.166702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.166681    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.188590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.188555    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a0f7e-openshift-default-dockercfg-jm67d\""
Oct 30 03:21:49.195031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.194995    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:21:49.244509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244475    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244528    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244599    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244653    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244696    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244779    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.244918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.245028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.244954    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wj9\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.245083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.245067    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.348345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.348293    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.353378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.353342    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.353714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.353689    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.385528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.385491    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.385824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.385801    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.386006 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.385987    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.386189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.386175    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wj9\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.386446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.386429    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.386679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.386664    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.387640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.387612    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.387792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.357049    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.387895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.365906    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.388596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.388571    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.423304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.390250    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.423304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.390271    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.423304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.405769    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.447254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.425222    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wj9\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.447254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.432744    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\") pod \"e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") " pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.491255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.490742    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:49.897353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.897321    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:21:49.897627 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:21:49.897603    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb60bb_aae7_469c_a7f5_015f5a610c1f.slice/crio-84f8396f618fb236371c2672af1a39e69b4d63912b067f8873824961df715ed0 WatchSource:0}: Error finding container 84f8396f618fb236371c2672af1a39e69b4d63912b067f8873824961df715ed0: Status 404 returned error can't find the container with id 84f8396f618fb236371c2672af1a39e69b4d63912b067f8873824961df715ed0
Oct 30 03:21:49.900999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.900965    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:49.901118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.901044    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:49.901118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.901088    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:21:49.996134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.996099    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerStarted","Data":"5a3ca86dfb6a0bcfe1cbe3ab2d85ef7fbe13b76938bd42d357d7bc0e125738ff"}
Oct 30 03:21:49.996330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:49.996151    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerStarted","Data":"84f8396f618fb236371c2672af1a39e69b4d63912b067f8873824961df715ed0"}
Oct 30 03:21:50.043844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:50.043789    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podStartSLOduration=1.043774885 podStartE2EDuration="1.043774885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:50.041608452 +0000 UTC m=+2449.115779285" watchObservedRunningTime="2026-10-30 03:21:50.043774885 +0000 UTC m=+2449.117945678"
Oct 30 03:21:50.492152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:50.492118    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:50.498461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:50.498437    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:51.005210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:51.005177    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:51.006078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:51.006054    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:21:51.632689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:51.632655    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:21:51.633200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:51.632858    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy" containerID="cri-o://a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba" gracePeriod=30
Oct 30 03:21:54.236286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.236248    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:21:54.240401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.240377    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.261811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.261780    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:21:54.273555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.273521    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a0f7e\""
Oct 30 03:21:54.372279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.370327    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtqf\" (UniqueName: \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.372279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.370726    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.481257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.480887    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtqf\" (UniqueName: \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.481468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.481287    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.494298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.483416    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.516251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.513972    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtqf\" (UniqueName: \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\") pod \"payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.560254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.559812    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:54.742546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.731773    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:21:54.790706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:54.790029    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:21:55.055341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:55.055200    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:21:55.083041 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:55.082996    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" event={"ID":"36113c7c-5e08-46ca-af47-57bb5cd9ae9a","Type":"ContainerStarted","Data":"609bf34d8dfa171b7e232e649dcda89c222955b255684808a0c603377cef6c87"}
Oct 30 03:21:56.091526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.091482    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" event={"ID":"36113c7c-5e08-46ca-af47-57bb5cd9ae9a","Type":"ContainerStarted","Data":"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"}
Oct 30 03:21:56.091944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.091592    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:21:56.939797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.939769    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:21:56.995117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995078    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995368    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995417    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:56.995674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995583    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995623    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995672    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995698    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:56.995870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995701    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.995870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.995766    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.996083    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.996262    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.996319    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\") pod \"8c7e7371-b50b-40a7-be15-1d852d292aa4\" (UID: \"8c7e7371-b50b-40a7-be15-1d852d292aa4\") "
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.996881    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.997265    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.997292    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.997307    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:56.997572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.997341    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.000740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:56.999996    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:57.000740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.000587    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" podStartSLOduration=3.000571668 podStartE2EDuration="3.000571668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:56.133720041 +0000 UTC m=+2455.207890837" watchObservedRunningTime="2026-10-30 03:21:57.000571668 +0000 UTC m=+2456.074742461"
Oct 30 03:21:57.001456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.001435    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:57.002850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.002806    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "kube-api-access-554h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:57.003580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.003440    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:57.003580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.003469    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token" pod "8c7e7371-b50b-40a7-be15-1d852d292aa4" (UID: "8c7e7371-b50b-40a7-be15-1d852d292aa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:57.098623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.098595    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.098623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.098618    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7e7371-b50b-40a7-be15-1d852d292aa4-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.098623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.098627    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.099280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.098636    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.099280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.098644    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554h4\" (UniqueName: \"kubernetes.io/projected/8c7e7371-b50b-40a7-be15-1d852d292aa4-kube-api-access-554h4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:21:57.101654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.101622    4214 generic.go:356] "Generic (PLEG): container finished" podID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerID="a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba" exitCode=0
Oct 30 03:21:57.101789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.101710    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"
Oct 30 03:21:57.101789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.101742    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerDied","Data":"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"}
Oct 30 03:21:57.101914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.101809    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz" event={"ID":"8c7e7371-b50b-40a7-be15-1d852d292aa4","Type":"ContainerDied","Data":"bc18e2967786fd2150941248f65289b0cd8cd233ad4d4376ed85f9b89ae1a767"}
Oct 30 03:21:57.101914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.101843    4214 scope.go:122] "RemoveContainer" containerID="a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"
Oct 30 03:21:57.113079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.113060    4214 scope.go:122] "RemoveContainer" containerID="24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"
Oct 30 03:21:57.123837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.123821    4214 scope.go:122] "RemoveContainer" containerID="a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"
Oct 30 03:21:57.124103 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:57.124083    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba\": container with ID starting with a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba not found: ID does not exist" containerID="a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"
Oct 30 03:21:57.124178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.124116    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba"} err="failed to get container status \"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba\": rpc error: code = NotFound desc = could not find container \"a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba\": container with ID starting with a19d7218c709f3423b1a00317801b617be7ed2b814fa6b7a33a8a7904a740aba not found: ID does not exist"
Oct 30 03:21:57.124178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.124134    4214 scope.go:122] "RemoveContainer" containerID="24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"
Oct 30 03:21:57.124420 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:21:57.124399    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c\": container with ID starting with 24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c not found: ID does not exist" containerID="24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"
Oct 30 03:21:57.124469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.124425    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c"} err="failed to get container status \"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c\": rpc error: code = NotFound desc = could not find container \"24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c\": container with ID starting with 24eb3864347d778490ab1d983befba226672023f8987b6b6c544b8e1e067014c not found: ID does not exist"
Oct 30 03:21:57.165734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.165665    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:21:57.175996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:57.175967    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b0b40d-openshift-default-7445b96d99-rjmcz"]
Oct 30 03:21:59.072447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:59.072406    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" path="/var/lib/kubelet/pods/8c7e7371-b50b-40a7-be15-1d852d292aa4/volumes"
Oct 30 03:21:59.732569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:59.732523    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" podUID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" containerName="payload-processing" containerID="cri-o://3bdf5a054ec8ed29915ff667962f44e8d9945429a50917f2ed81035bf239b998" gracePeriod=25
Oct 30 03:21:59.791268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:21:59.791189    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" podUID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" containerName="payload-pre-processing" containerID="cri-o://99525bf0eadfab529543ebc9899d701cceca00c5e008d11a1af817091a38918a" gracePeriod=25
Oct 30 03:22:00.128333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.128304    4214 generic.go:356] "Generic (PLEG): container finished" podID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" containerID="3bdf5a054ec8ed29915ff667962f44e8d9945429a50917f2ed81035bf239b998" exitCode=0
Oct 30 03:22:00.128747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.128374    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" event={"ID":"ef02104c-7af8-4fab-9cb1-7c09afeb1f77","Type":"ContainerDied","Data":"3bdf5a054ec8ed29915ff667962f44e8d9945429a50917f2ed81035bf239b998"}
Oct 30 03:22:00.128747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.128430    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn" event={"ID":"ef02104c-7af8-4fab-9cb1-7c09afeb1f77","Type":"ContainerDied","Data":"0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d"}
Oct 30 03:22:00.128747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.128449    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5988f9e9c370b2371eb62b9799f20ad59f958981face715c19a11cfa57453d"
Oct 30 03:22:00.130038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.130019    4214 generic.go:356] "Generic (PLEG): container finished" podID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" containerID="99525bf0eadfab529543ebc9899d701cceca00c5e008d11a1af817091a38918a" exitCode=0
Oct 30 03:22:00.130152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.130089    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" event={"ID":"acdda8fe-f076-48dd-bd2b-9fabc523dd1b","Type":"ContainerDied","Data":"99525bf0eadfab529543ebc9899d701cceca00c5e008d11a1af817091a38918a"}
Oct 30 03:22:00.130152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.130137    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2" event={"ID":"acdda8fe-f076-48dd-bd2b-9fabc523dd1b","Type":"ContainerDied","Data":"b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d"}
Oct 30 03:22:00.130271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.130153    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73a1d6bbd39cb73873161e300d3d3811dc4a9924eaec9f81eeb2e82dd7eba4d"
Oct 30 03:22:00.136704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.136683    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:22:00.141727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.141705    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:22:00.256094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.255981    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\") pod \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") "
Oct 30 03:22:00.256616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.256588    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\") pod \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") "
Oct 30 03:22:00.256741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.256688    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\" (UniqueName: \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\") pod \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\" (UID: \"acdda8fe-f076-48dd-bd2b-9fabc523dd1b\") "
Oct 30 03:22:00.256741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.256723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\" (UniqueName: \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\") pod \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\" (UID: \"ef02104c-7af8-4fab-9cb1-7c09afeb1f77\") "
Oct 30 03:22:00.257526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.256447    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume" pod "acdda8fe-f076-48dd-bd2b-9fabc523dd1b" (UID: "acdda8fe-f076-48dd-bd2b-9fabc523dd1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:00.257847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.257667    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:00.257847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.257807    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume" pod "ef02104c-7af8-4fab-9cb1-7c09afeb1f77" (UID: "ef02104c-7af8-4fab-9cb1-7c09afeb1f77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:00.260190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.260162    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt" pod "ef02104c-7af8-4fab-9cb1-7c09afeb1f77" (UID: "ef02104c-7af8-4fab-9cb1-7c09afeb1f77"). InnerVolumeSpecName "kube-api-access-qzltt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:00.260323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.260299    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn" pod "acdda8fe-f076-48dd-bd2b-9fabc523dd1b" (UID: "acdda8fe-f076-48dd-bd2b-9fabc523dd1b"). InnerVolumeSpecName "kube-api-access-fk8sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:00.359630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.359592    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:00.359630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.359631    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8sn\" (UniqueName: \"kubernetes.io/projected/acdda8fe-f076-48dd-bd2b-9fabc523dd1b-kube-api-access-fk8sn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:00.359860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:00.359648    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzltt\" (UniqueName: \"kubernetes.io/projected/ef02104c-7af8-4fab-9cb1-7c09afeb1f77-kube-api-access-qzltt\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:01.145353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.145318    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"
Oct 30 03:22:01.146408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.146384    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"
Oct 30 03:22:01.220297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.220262    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:22:01.232923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.232893    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-7cdf4b996d-ntfgn"]
Oct 30 03:22:01.259107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.259073    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:22:01.263619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:01.263587    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-68c884f987-fsmc2"]
Oct 30 03:22:03.070892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:03.070851    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" path="/var/lib/kubelet/pods/acdda8fe-f076-48dd-bd2b-9fabc523dd1b/volumes"
Oct 30 03:22:03.071437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:03.071402    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" path="/var/lib/kubelet/pods/ef02104c-7af8-4fab-9cb1-7c09afeb1f77/volumes"
Oct 30 03:22:04.485850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.485813    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:22:04.487837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487800    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.487837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487829    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487845    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487851    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487859    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" containerName="payload-processing"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487865    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" containerName="payload-processing"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487901    4214 cpu_manager.go:414] "containerMap: removing container" podUID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" containerName="payload-pre-processing"
Oct 30 03:22:04.488037 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.487906    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" containerName="payload-pre-processing"
Oct 30 03:22:04.488424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.488053    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdda8fe-f076-48dd-bd2b-9fabc523dd1b" containerName="payload-pre-processing"
Oct 30 03:22:04.488424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.488066    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.488424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.488074    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7e7371-b50b-40a7-be15-1d852d292aa4" containerName="istio-proxy"
Oct 30 03:22:04.488424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.488081    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef02104c-7af8-4fab-9cb1-7c09afeb1f77" containerName="payload-processing"
Oct 30 03:22:04.493290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.493268    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.505760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.505736    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:22:04.519282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.515576    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-874ab5-openshift-default-dockercfg-kmjhl\""
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.551774    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.551916    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.551992    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552050    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552077    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmpr\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552181    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552296    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552385    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.552743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.552410    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656287    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656344    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656481    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656544    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656611    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656663    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656689    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmpr\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656791    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.656859    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.657744    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.658253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.658023    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.659048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.658652    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.659048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.658898    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.687254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.686916    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.779295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.735453    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.779295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.769087    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.780701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.780186    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.795265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.787483    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmpr\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:04.814467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:04.814436    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:05.270662 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:22:05.270630    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8da9c6_da55_45d6_9f70_f71f00866b63.slice/crio-91f8ad99ead6559180db7c17b1b48c2be1e5da51bc13411be2777fc1d8ae24f2 WatchSource:0}: Error finding container 91f8ad99ead6559180db7c17b1b48c2be1e5da51bc13411be2777fc1d8ae24f2: Status 404 returned error can't find the container with id 91f8ad99ead6559180db7c17b1b48c2be1e5da51bc13411be2777fc1d8ae24f2
Oct 30 03:22:05.273862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:05.273832    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:22:05.274496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:05.274465    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:05.274602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:05.274542    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:05.274602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:05.274572    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:06.232929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:06.232890    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerStarted","Data":"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"}
Oct 30 03:22:06.233414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:06.232955    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerStarted","Data":"91f8ad99ead6559180db7c17b1b48c2be1e5da51bc13411be2777fc1d8ae24f2"}
Oct 30 03:22:06.296793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:06.296741    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podStartSLOduration=2.29672677 podStartE2EDuration="2.29672677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:06.294129047 +0000 UTC m=+2465.368299841" watchObservedRunningTime="2026-10-30 03:22:06.29672677 +0000 UTC m=+2465.370897562"
Oct 30 03:22:06.819308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:06.819270    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:06.825558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:06.825531    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:07.106036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:07.106006    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:22:07.243311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:07.243271    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:07.244391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:07.244365    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:22:09.168981 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:09.168935    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:22:10.681850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.681773    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:22:10.689476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.689453    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:10.706779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.706747    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:22:10.711978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.711949    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:10.712355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.712328    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjfd\" (UniqueName: \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:10.713817 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.713795    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-874ab5-dockercfg-gb76v\""
Oct 30 03:22:10.714117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.714066    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-874ab5\""
Oct 30 03:22:10.762251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.757758    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:22:10.764258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.764140    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.857798    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjfd\" (UniqueName: \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.857896    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.857931    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlq58\" (UniqueName: \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.858001    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.858934    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.979358    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.979414    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlq58\" (UniqueName: \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.011164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:10.981890    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.032637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.032600    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:22:11.044679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.044603    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjfd\" (UniqueName: \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\") pod \"payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:11.047419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.047392    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlq58\" (UniqueName: \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\") pod \"payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.312469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.312436    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:11.320772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.320746    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:11.838167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.838137    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:22:11.852702 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:22:11.852676    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c10691e_c6f6_4ccc_b5e2_4155217cca9a.slice/crio-02b6e4b51538bd6edb957a6e7f9c6f91829186b752f65a11e0764cfdc74555f5 WatchSource:0}: Error finding container 02b6e4b51538bd6edb957a6e7f9c6f91829186b752f65a11e0764cfdc74555f5: Status 404 returned error can't find the container with id 02b6e4b51538bd6edb957a6e7f9c6f91829186b752f65a11e0764cfdc74555f5
Oct 30 03:22:11.856803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:11.856772    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:22:12.288896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.288795    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" event={"ID":"7c10691e-c6f6-4ccc-b5e2-4155217cca9a","Type":"ContainerStarted","Data":"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"}
Oct 30 03:22:12.288896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.288852    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" event={"ID":"7c10691e-c6f6-4ccc-b5e2-4155217cca9a","Type":"ContainerStarted","Data":"02b6e4b51538bd6edb957a6e7f9c6f91829186b752f65a11e0764cfdc74555f5"}
Oct 30 03:22:12.288896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.288892    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:12.291189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.291158    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" event={"ID":"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c","Type":"ContainerStarted","Data":"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"}
Oct 30 03:22:12.291372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.291207    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" event={"ID":"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c","Type":"ContainerStarted","Data":"6d9aa88f16be69cadfbd64a34aeede0425249ddbb6cb1ecc63982a95f65de38f"}
Oct 30 03:22:12.291372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.291276    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:12.331477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.331433    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" podStartSLOduration=2.331419659 podStartE2EDuration="2.331419659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:12.330089404 +0000 UTC m=+2471.404260198" watchObservedRunningTime="2026-10-30 03:22:12.331419659 +0000 UTC m=+2471.405590451"
Oct 30 03:22:12.393349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:12.393292    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" podStartSLOduration=2.393278852 podStartE2EDuration="2.393278852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:12.387640257 +0000 UTC m=+2471.461811050" watchObservedRunningTime="2026-10-30 03:22:12.393278852 +0000 UTC m=+2471.467449645"
Oct 30 03:22:22.314788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:22.314382    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:22:22.314788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:22.314457    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:23.381965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:23.381930    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:22:23.383560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:23.383537    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:22:26.715763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.715730    4214 generic.go:356] "Generic (PLEG): container finished" podID="f592998c-cb32-4292-a4ac-86eca80e305e" containerID="9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c" exitCode=0
Oct 30 03:22:26.716334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.715801    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerDied","Data":"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"}
Oct 30 03:22:26.716542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.716515    4214 scope.go:122] "RemoveContainer" containerID="9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"
Oct 30 03:22:26.718635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.718601    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:26.718740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.718695    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:26.718809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:26.718747    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:27.727893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:27.727860    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerStarted","Data":"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"}
Oct 30 03:22:28.658251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:28.657250    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:22:28.658251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:28.657562    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 03:22:28.658251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:28.657614    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 03:22:29.655662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:29.655619    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 03:22:29.656153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:29.655685    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 03:22:30.661665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:30.661618    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 03:22:30.662155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:30.661685    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 03:22:31.700307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:31.696192    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 03:22:31.717360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:31.717316    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 03:22:32.664379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.664340    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 03:22:32.665630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.664420    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 03:22:32.972500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.972360    4214 generic.go:356] "Generic (PLEG): container finished" podID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerID="e0fd857a00ec941296244fb4e4886bcaeb3445c75bc9cebd48cc6a58aebb757e" exitCode=0
Oct 30 03:22:32.973109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.972523    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerDied","Data":"e0fd857a00ec941296244fb4e4886bcaeb3445c75bc9cebd48cc6a58aebb757e"}
Oct 30 03:22:32.973789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.973766    4214 scope.go:122] "RemoveContainer" containerID="e0fd857a00ec941296244fb4e4886bcaeb3445c75bc9cebd48cc6a58aebb757e"
Oct 30 03:22:32.976428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.975875    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:32.976428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.976064    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:32.976428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:32.976100    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:33.658711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:33.658674    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:22:33.659159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:33.659138    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:22:33.659894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:33.659875    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:22:33.990680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:33.990591    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerStarted","Data":"8c017575d9c3bc2b1a309bc6124c8080e8858334aa11a3d02229b304a38c2133"}
Oct 30 03:22:34.765098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:34.765057    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:22:34.768256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:34.766366    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 03:22:34.768256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:34.766423    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 03:22:35.271478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.271445    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:22:35.305071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.297143    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.305071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.304534    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:22:35.320475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.320444    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5rj\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.320664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.320646    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.320878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.320847    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.320980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.320901    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.320980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.320929    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.321067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.321005    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.321155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.321140    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.321208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.321178    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.321315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.321297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.422682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422648    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.422877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422695    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.422877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422781    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.422877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422865    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5rj\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422897    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.422985    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423036    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423069    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423083    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423133    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423164    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423171    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.423498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423444    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.424026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.423993    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.425371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.425348    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.425504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.425486    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.468125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.468093    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.470365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.470340    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5rj\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.637253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.635984    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:35.769339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.767638    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 03:22:35.769339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:35.767707    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 03:22:36.108427 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:22:36.108390    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f75a085_ce38_40e8_be56_a8ddaebf1336.slice/crio-531b548efe405eb3b5eb7387e1dc577778a68c642f59e2fc2dba5386d50cabb8 WatchSource:0}: Error finding container 531b548efe405eb3b5eb7387e1dc577778a68c642f59e2fc2dba5386d50cabb8: Status 404 returned error can't find the container with id 531b548efe405eb3b5eb7387e1dc577778a68c642f59e2fc2dba5386d50cabb8
Oct 30 03:22:36.108734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.108713    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:22:36.111259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.111199    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:36.111360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.111285    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:36.111360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.111315    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:22:36.766667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.766629    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 03:22:36.767332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:36.767299    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 03:22:37.097983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.097675    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" event={"ID":"7f75a085-ce38-40e8-be56-a8ddaebf1336","Type":"ContainerStarted","Data":"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"}
Oct 30 03:22:37.097983 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.097740    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" event={"ID":"7f75a085-ce38-40e8-be56-a8ddaebf1336","Type":"ContainerStarted","Data":"531b548efe405eb3b5eb7387e1dc577778a68c642f59e2fc2dba5386d50cabb8"}
Oct 30 03:22:37.202027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.201968    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podStartSLOduration=2.201950126 podStartE2EDuration="2.201950126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:37.160740638 +0000 UTC m=+2496.234911433" watchObservedRunningTime="2026-10-30 03:22:37.201950126 +0000 UTC m=+2496.276120919"
Oct 30 03:22:37.685253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.684279    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:37.705254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.686569    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:37.705254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.686741    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:37.789280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.774363    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 03:22:37.789280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:37.774418    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 03:22:38.652378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:38.651614    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:38.652378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:38.651679    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:38.726834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:38.726801    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:22:38.769256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:38.768211    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body=
Oct 30 03:22:38.769256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:38.768286    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused"
Oct 30 03:22:39.230406 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:39.230361    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:22:39.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:39.651578    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:39.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:39.651639    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:39.771390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:39.771355    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:22:39.772622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:39.772600    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:22:40.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:40.652657    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:40.657254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:40.652720    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:41.671698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:41.671656    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:41.672216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:41.671724    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:41.673125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:41.673088    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:22:42.666689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:42.666639    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Oct 30 03:22:42.666918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:42.666700    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Oct 30 03:22:43.657072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:43.657030    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:43.657584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:43.657480    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:43.658439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:43.658416    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:22:45.858925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:45.858885    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:22:45.859471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:45.859193    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy" containerID="cri-o://7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba" gracePeriod=30
Oct 30 03:22:51.187699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.187629    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:22:51.199177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199151    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199356    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199423    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199480    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199653    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199726    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199773    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.199823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199806    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199817    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.200030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199869    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:51.200030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.199874    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\") pod \"931d8eb0-6f34-4a25-886d-851eff161cd0\" (UID: \"931d8eb0-6f34-4a25-886d-851eff161cd0\") "
Oct 30 03:22:51.200186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200114    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200398    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200592    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.200752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200736    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.200838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200758    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931d8eb0-6f34-4a25-886d-851eff161cd0-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.200838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200772    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.200838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200787    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.200838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.200799    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.211404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.202482    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:51.211404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.204736    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:22:51.211404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.204810    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "kube-api-access-8cc9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:51.211404 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.205076    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token" pod "931d8eb0-6f34-4a25-886d-851eff161cd0" (UID: "931d8eb0-6f34-4a25-886d-851eff161cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:51.302743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.302704    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.302738    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc9r\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-kube-api-access-8cc9r\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.302743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.302750    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.303000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.302759    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931d8eb0-6f34-4a25-886d-851eff161cd0-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:51.324438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.324407    4214 generic.go:356] "Generic (PLEG): container finished" podID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerID="7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba" exitCode=0
Oct 30 03:22:51.324617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.324469    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"
Oct 30 03:22:51.324617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.324472    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerDied","Data":"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"}
Oct 30 03:22:51.324617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.324596    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m" event={"ID":"931d8eb0-6f34-4a25-886d-851eff161cd0","Type":"ContainerDied","Data":"0cfc1bacd53c64cc3c1a126e97e050b89a684f81ad8892323e01a3ae26ab91bf"}
Oct 30 03:22:51.324766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.324619    4214 scope.go:122] "RemoveContainer" containerID="7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"
Oct 30 03:22:51.343693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.343669    4214 scope.go:122] "RemoveContainer" containerID="77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"
Oct 30 03:22:51.362383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.362195    4214 scope.go:122] "RemoveContainer" containerID="7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"
Oct 30 03:22:51.362863 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:51.362830    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba\": container with ID starting with 7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba not found: ID does not exist" containerID="7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"
Oct 30 03:22:51.362974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.362871    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba"} err="failed to get container status \"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba\": rpc error: code = NotFound desc = could not find container \"7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba\": container with ID starting with 7d3d2f54aea9e44dd0a2910b3824f0590b14bf3a74dae0f579852f1ce5064bba not found: ID does not exist"
Oct 30 03:22:51.362974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.362897    4214 scope.go:122] "RemoveContainer" containerID="77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"
Oct 30 03:22:51.363551 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:51.363515    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f\": container with ID starting with 77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f not found: ID does not exist" containerID="77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"
Oct 30 03:22:51.363657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.363558    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f"} err="failed to get container status \"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f\": rpc error: code = NotFound desc = could not find container \"77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f\": container with ID starting with 77a91baeed00fc5ea24f6ffa52f39eb803f358319f4df4f81dda16781a4f186f not found: ID does not exist"
Oct 30 03:22:51.389403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.389377    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:22:51.394819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:51.394797    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4cf8f5-openshift-default-867459b9-9ww7m"]
Oct 30 03:22:53.072191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.072149    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" path="/var/lib/kubelet/pods/931d8eb0-6f34-4a25-886d-851eff161cd0/volumes"
Oct 30 03:22:53.777756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.777719    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:22:53.780026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.779994    4214 cpu_manager.go:414] "containerMap: removing container" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:53.784775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.784750    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:53.786072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.786048    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:53.786287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.786274    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:53.807424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.807397    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:22:53.807788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.807567    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:53.825186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.825164    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-4b5459-dockercfg-ptpxw\""
Oct 30 03:22:53.839292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.839269    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-4b5459\""
Oct 30 03:22:53.912247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.912193    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbf9\" (UniqueName: \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:53.912441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:53.912347    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.014039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.013996    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbf9\" (UniqueName: \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.014278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.014117    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.014966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.014909    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.035259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.034634    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbf9\" (UniqueName: \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\") pod \"payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.125589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.125554    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:54.167698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.167659    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:22:54.168547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.168524    4214 cpu_manager.go:414] "containerMap: removing container" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:54.168547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.168545    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931d8eb0-6f34-4a25-886d-851eff161cd0" containerName="istio-proxy"
Oct 30 03:22:54.172835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.172814    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.200000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.197328    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:22:54.208423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.207267    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-874ab5-serving-cert\""
Oct 30 03:22:54.225684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.222124    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.225684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.222153    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzkpq\" (UniqueName: \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.225684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.222253    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.225684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.222352    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.324189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.324156    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.324369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.324199    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzkpq\" (UniqueName: \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.324369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.324320    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.324471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.324455    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.324554 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:54.324456    4214 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-serving-cert: secret "maas-api-e2e-shared-b-w0-874ab5-serving-cert" not found
Oct 30 03:22:54.324618 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:54.324607    4214 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls podName:ede07f81-f152-4233-af36-5af83c59dec3 nodeName:}" failed. No retries permitted until 2026-10-30 03:22:54.82458528 +0000 UTC m=+2513.898756051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls") pod "maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" (UID: "ede07f81-f152-4233-af36-5af83c59dec3") : secret "maas-api-e2e-shared-b-w0-874ab5-serving-cert" not found
Oct 30 03:22:54.325031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.325010    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.326855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.326835    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.367260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.357030    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzkpq\" (UniqueName: \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.480081 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:22:54.480041    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9307f6b3_02e6_43bf_9224_c6256bef88f8.slice/crio-50acf89cc42d04eab4bbc402783c475246c1804953251d94200c9335e3c7403f WatchSource:0}: Error finding container 50acf89cc42d04eab4bbc402783c475246c1804953251d94200c9335e3c7403f: Status 404 returned error can't find the container with id 50acf89cc42d04eab4bbc402783c475246c1804953251d94200c9335e3c7403f
Oct 30 03:22:54.482447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.482414    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:22:54.832616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.832580    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:54.835521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:54.835496    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:55.104547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.104461    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:55.421905 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:22:55.421873    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede07f81_f152_4233_af36_5af83c59dec3.slice/crio-71c8811285eff4f731c34704e877d9690062ff429d660b3c8bcdcdcc9a806269 WatchSource:0}: Error finding container 71c8811285eff4f731c34704e877d9690062ff429d660b3c8bcdcdcc9a806269: Status 404 returned error can't find the container with id 71c8811285eff4f731c34704e877d9690062ff429d660b3c8bcdcdcc9a806269
Oct 30 03:22:55.422667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.422638    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:22:55.429317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.429284    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" event={"ID":"9307f6b3-02e6-43bf-9224-c6256bef88f8","Type":"ContainerStarted","Data":"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"}
Oct 30 03:22:55.429474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.429342    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" event={"ID":"9307f6b3-02e6-43bf-9224-c6256bef88f8","Type":"ContainerStarted","Data":"50acf89cc42d04eab4bbc402783c475246c1804953251d94200c9335e3c7403f"}
Oct 30 03:22:55.429474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.429420    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:22:55.481250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:55.478096    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" podStartSLOduration=2.4780770309999998 podStartE2EDuration="2.478077031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:55.477699841 +0000 UTC m=+2514.551870635" watchObservedRunningTime="2026-10-30 03:22:55.478077031 +0000 UTC m=+2514.552247826"
Oct 30 03:22:56.552589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.552445    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" event={"ID":"ede07f81-f152-4233-af36-5af83c59dec3","Type":"ContainerStarted","Data":"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"}
Oct 30 03:22:56.552589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.552509    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" event={"ID":"ede07f81-f152-4233-af36-5af83c59dec3","Type":"ContainerStarted","Data":"71c8811285eff4f731c34704e877d9690062ff429d660b3c8bcdcdcc9a806269"}
Oct 30 03:22:56.552589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.552538    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:22:56.597614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.597556    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" podStartSLOduration=2.2120993110000002 podStartE2EDuration="2.597540722s" totalImagesPullingTime="385.441411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:56.594999663 +0000 UTC m=+2515.669170465" watchObservedRunningTime="2026-10-30 03:22:56.597540722 +0000 UTC m=+2515.671711568"
Oct 30 03:22:56.705371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.705331    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerName="main" containerID="cri-o://63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb" gracePeriod=45
Oct 30 03:22:56.727769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:56.725856    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerName="main" probeResult="failure" output="Get \"https://10.133.0.252:8000/ready\": dial tcp 10.133.0.252:8000: connect: connection refused"
Oct 30 03:22:57.010497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.010466    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:22:57.089213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.088806    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.089213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.088875    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089275    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\" (UniqueName: \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089316    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089361    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089414    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089448    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.089992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.089640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\") pod \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\" (UID: \"bef7941a-6f8b-45ec-b2cb-5e2b892e0164\") "
Oct 30 03:22:57.091900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.091872    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:57.092409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.092386    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.092409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.092409    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.092574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.092423    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.098640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.098611    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:57.104026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.103995    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:57.108587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.107315    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq" pod "bef7941a-6f8b-45ec-b2cb-5e2b892e0164" (UID: "bef7941a-6f8b-45ec-b2cb-5e2b892e0164"). InnerVolumeSpecName "kube-api-access-n7vxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:57.193782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.193744    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7vxq\" (UniqueName: \"kubernetes.io/projected/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-kube-api-access-n7vxq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.193782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.193777    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.193782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.193788    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7941a-6f8b-45ec-b2cb-5e2b892e0164-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:22:57.565624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.565590    4214 generic.go:356] "Generic (PLEG): container finished" podID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerID="63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb" exitCode=0
Oct 30 03:22:57.566062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.565670    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"
Oct 30 03:22:57.566062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.565675    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" event={"ID":"bef7941a-6f8b-45ec-b2cb-5e2b892e0164","Type":"ContainerDied","Data":"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"}
Oct 30 03:22:57.566062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.565730    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp" event={"ID":"bef7941a-6f8b-45ec-b2cb-5e2b892e0164","Type":"ContainerDied","Data":"d10be56b0261b9c245a4b34ccf7af11aebbc6c0fe34032b130a6832ef9a6a1e3"}
Oct 30 03:22:57.566062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.565755    4214 scope.go:122] "RemoveContainer" containerID="63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"
Oct 30 03:22:57.579476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.579456    4214 scope.go:122] "RemoveContainer" containerID="63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"
Oct 30 03:22:57.579826 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:22:57.579802    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb\": container with ID starting with 63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb not found: ID does not exist" containerID="63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"
Oct 30 03:22:57.579891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.579838    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb"} err="failed to get container status \"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb\": rpc error: code = NotFound desc = could not find container \"63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb\": container with ID starting with 63010b8c648a4d051f60c55b001ecf6001d4a3d75491aae0bd4334306ee42cfb not found: ID does not exist"
Oct 30 03:22:57.622292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.622266    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:22:57.635005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:57.634981    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9245d1/rate-test-model-9245d1-kserve-54f9f9b745-j2tpp"]
Oct 30 03:22:59.070839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:22:59.070801    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" path="/var/lib/kubelet/pods/bef7941a-6f8b-45ec-b2cb-5e2b892e0164/volumes"
Oct 30 03:23:02.572322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:02.572292    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:23:06.098112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:06.098073    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:23:06.576693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:06.576651    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:23:09.494164 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:23:09.494125    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:23:14.902792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.902759    4214 generic.go:356] "Generic (PLEG): container finished" podID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerID="5a3ca86dfb6a0bcfe1cbe3ab2d85ef7fbe13b76938bd42d357d7bc0e125738ff" exitCode=0
Oct 30 03:23:14.903291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.902800    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerDied","Data":"5a3ca86dfb6a0bcfe1cbe3ab2d85ef7fbe13b76938bd42d357d7bc0e125738ff"}
Oct 30 03:23:14.903352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.903310    4214 scope.go:122] "RemoveContainer" containerID="5a3ca86dfb6a0bcfe1cbe3ab2d85ef7fbe13b76938bd42d357d7bc0e125738ff"
Oct 30 03:23:14.905438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.905401    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:23:14.905544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.905486    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:23:14.905595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:14.905541    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:23:15.913316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:15.913281    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerStarted","Data":"66c1729bc589a7a60fc4d748d0b3340b7952cac8ba83aedee840e5b0b13886f8"}
Oct 30 03:23:16.492102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:16.492062    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:23:16.493171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:16.493140    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 03:23:16.493353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:16.493201    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 03:23:17.180286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:17.180248    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:23:17.503303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:17.495570    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 03:23:17.503303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:17.495640    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 03:23:18.492074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:18.492039    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 03:23:18.492514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:18.492109    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 03:23:19.492167 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:19.492130    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 03:23:19.492607 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:19.492181    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 03:23:20.492174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:20.492135    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 03:23:20.492778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:20.492743    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 03:23:21.005327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.005295    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:23:21.127904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.127846    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerName="main" containerID="cri-o://e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc" gracePeriod=45
Oct 30 03:23:21.223992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.223953    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:23:21.224440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.224410    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" containerName="maas-api" containerID="cri-o://15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda" gracePeriod=30
Oct 30 03:23:21.285327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.285015    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": dial tcp 10.133.1.2:8000: connect: connection refused"
Oct 30 03:23:21.396141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.391997    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:23:21.396141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.392262    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" podUID="1991f52c-7032-403a-8ab0-6e02949aa979" containerName="payload-processing" containerID="cri-o://62671c77e0ac7fea83876eb8aa8c70f84db039de44365e02eb51cbf215828e14" gracePeriod=30
Oct 30 03:23:21.460206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.431546    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:23:21.460206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.431796    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" podUID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" containerName="payload-pre-processing" containerID="cri-o://df8f8ceedfcffe7ec2a6856b00bfe8f5b9b0ff3833ded492814e5f38bbd4bfbb" gracePeriod=30
Oct 30 03:23:21.574855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.574826    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:23:21.643745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.643692    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794083    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\" (UniqueName: \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794249    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794293    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794341    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794412    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.794537    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\") pod \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\" (UID: \"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0\") "
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.819329    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.819631    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:21.820559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.819907    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:21.951513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.921923    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.951513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.921960    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.951513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.921974    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.971424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.960205    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:21.971424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.960895    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "kube-api-access-6gwpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:21.971424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:21.964674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs" pod "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" (UID: "2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:22.004212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.004181    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:23:22.031794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.031475    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwpv\" (UniqueName: \"kubernetes.io/projected/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-kube-api-access-6gwpv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.031794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.031509    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.031794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.031523    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.035784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.035210    4214 generic.go:356] "Generic (PLEG): container finished" podID="1a86ecbd-0a65-4736-9b70-669c92109124" containerID="15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda" exitCode=0
Oct 30 03:23:22.035784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.035367    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" event={"ID":"1a86ecbd-0a65-4736-9b70-669c92109124","Type":"ContainerDied","Data":"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"}
Oct 30 03:23:22.035784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.035422    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" event={"ID":"1a86ecbd-0a65-4736-9b70-669c92109124","Type":"ContainerDied","Data":"504c7bbf8ff76320f062a8a55c19995fad864e999f7c6deba22560dd98726b9d"}
Oct 30 03:23:22.035784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.035456    4214 scope.go:122] "RemoveContainer" containerID="15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"
Oct 30 03:23:22.035784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.035632    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"
Oct 30 03:23:22.052766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.052738    4214 generic.go:356] "Generic (PLEG): container finished" podID="1991f52c-7032-403a-8ab0-6e02949aa979" containerID="62671c77e0ac7fea83876eb8aa8c70f84db039de44365e02eb51cbf215828e14" exitCode=0
Oct 30 03:23:22.054821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.052930    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" event={"ID":"1991f52c-7032-403a-8ab0-6e02949aa979","Type":"ContainerDied","Data":"62671c77e0ac7fea83876eb8aa8c70f84db039de44365e02eb51cbf215828e14"}
Oct 30 03:23:22.055025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.054669    4214 generic.go:356] "Generic (PLEG): container finished" podID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerID="e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc" exitCode=0
Oct 30 03:23:22.055321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.054729    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"
Oct 30 03:23:22.075887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.074504    4214 generic.go:356] "Generic (PLEG): container finished" podID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" containerID="df8f8ceedfcffe7ec2a6856b00bfe8f5b9b0ff3833ded492814e5f38bbd4bfbb" exitCode=0
Oct 30 03:23:22.075887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.075759    4214 scope.go:122] "RemoveContainer" containerID="15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"
Oct 30 03:23:22.076461 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:23:22.076435    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda\": container with ID starting with 15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda not found: ID does not exist" containerID="15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"
Oct 30 03:23:22.076650 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.076622    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda"} err="failed to get container status \"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda\": rpc error: code = NotFound desc = could not find container \"15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda\": container with ID starting with 15bc52933bca4a2fbad5abfc4f632f1c14bf6571a9eefb67c9074e73b156dcda not found: ID does not exist"
Oct 30 03:23:22.088098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.088065    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" event={"ID":"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0","Type":"ContainerDied","Data":"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"}
Oct 30 03:23:22.088244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.088202    4214 scope.go:122] "RemoveContainer" containerID="e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"
Oct 30 03:23:22.088353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.088204    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9" event={"ID":"2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0","Type":"ContainerDied","Data":"796eff80952ad486a5298eb2153a68654a73887fd86f673d0051e3662b797adf"}
Oct 30 03:23:22.088436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.088371    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" event={"ID":"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f","Type":"ContainerDied","Data":"df8f8ceedfcffe7ec2a6856b00bfe8f5b9b0ff3833ded492814e5f38bbd4bfbb"}
Oct 30 03:23:22.088748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.088570    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:23:22.105644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.105622    4214 scope.go:122] "RemoveContainer" containerID="e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"
Oct 30 03:23:22.106058 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:23:22.105979    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc\": container with ID starting with e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc not found: ID does not exist" containerID="e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"
Oct 30 03:23:22.106246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.106192    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc"} err="failed to get container status \"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc\": rpc error: code = NotFound desc = could not find container \"e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc\": container with ID starting with e8deb2ba0a0c9c0b7ede8be72ec58bb6060cce2d52e9e1cbff97cc8e07c273bc not found: ID does not exist"
Oct 30 03:23:22.133266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.133207    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\") pod \"1a86ecbd-0a65-4736-9b70-669c92109124\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") "
Oct 30 03:23:22.133420 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.133287    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\") pod \"1a86ecbd-0a65-4736-9b70-669c92109124\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") "
Oct 30 03:23:22.133502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.133480    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\") pod \"1a86ecbd-0a65-4736-9b70-669c92109124\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") "
Oct 30 03:23:22.133692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.133668    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\" (UniqueName: \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\") pod \"1a86ecbd-0a65-4736-9b70-669c92109124\" (UID: \"1a86ecbd-0a65-4736-9b70-669c92109124\") "
Oct 30 03:23:22.133821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.133795    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca" pod "1a86ecbd-0a65-4736-9b70-669c92109124" (UID: "1a86ecbd-0a65-4736-9b70-669c92109124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:22.139459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.134397    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a86ecbd-0a65-4736-9b70-669c92109124-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.141487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.141445    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls" pod "1a86ecbd-0a65-4736-9b70-669c92109124" (UID: "1a86ecbd-0a65-4736-9b70-669c92109124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:22.200030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.199340    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing" containerID="cri-o://ec16579a17b96b19147285f345e7d1d97e610e3a4ca18cbd1369a2127509475e" gracePeriod=25
Oct 30 03:23:22.203377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.203314    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k" pod "1a86ecbd-0a65-4736-9b70-669c92109124" (UID: "1a86ecbd-0a65-4736-9b70-669c92109124"). InnerVolumeSpecName "kube-api-access-tfb8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:22.203706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.203675    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs" pod "1a86ecbd-0a65-4736-9b70-669c92109124" (UID: "1a86ecbd-0a65-4736-9b70-669c92109124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:22.237725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.236780    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.237725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.236809    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfb8k\" (UniqueName: \"kubernetes.io/projected/1a86ecbd-0a65-4736-9b70-669c92109124-kube-api-access-tfb8k\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.237725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.236827    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a86ecbd-0a65-4736-9b70-669c92109124-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.237725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.237714    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:23:22.242619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.242595    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/rate-test-model-360818-kserve-564b476c45-5llx9"]
Oct 30 03:23:22.256373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.256356    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:23:22.262421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.262401    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:23:22.512051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.511787    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\" (UniqueName: \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\") pod \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") "
Oct 30 03:23:22.516399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.516333    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\") pod \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\" (UID: \"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f\") "
Oct 30 03:23:22.516399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.516396    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\" (UniqueName: \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\") pod \"1991f52c-7032-403a-8ab0-6e02949aa979\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") "
Oct 30 03:23:22.516730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.516604    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\") pod \"1991f52c-7032-403a-8ab0-6e02949aa979\" (UID: \"1991f52c-7032-403a-8ab0-6e02949aa979\") "
Oct 30 03:23:22.517380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.517324    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume" pod "fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" (UID: "fe390f2c-87b9-4f51-b0d6-59f185ac2c5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:22.517789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.517734    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume" pod "1991f52c-7032-403a-8ab0-6e02949aa979" (UID: "1991f52c-7032-403a-8ab0-6e02949aa979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:22.522267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.522195    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf" pod "1991f52c-7032-403a-8ab0-6e02949aa979" (UID: "1991f52c-7032-403a-8ab0-6e02949aa979"). InnerVolumeSpecName "kube-api-access-cj9kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:22.524005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.523982    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx" pod "fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" (UID: "fe390f2c-87b9-4f51-b0d6-59f185ac2c5f"). InnerVolumeSpecName "kube-api-access-rzjmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:22.584275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.584248    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:23:22.588569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.588545    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc"]
Oct 30 03:23:22.618907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.618877    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjmx\" (UniqueName: \"kubernetes.io/projected/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-kube-api-access-rzjmx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.618907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.618905    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.619056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.618915    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9kf\" (UniqueName: \"kubernetes.io/projected/1991f52c-7032-403a-8ab0-6e02949aa979-kube-api-access-cj9kf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.619056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:22.618923    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1991f52c-7032-403a-8ab0-6e02949aa979-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:23.069136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.069100    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" path="/var/lib/kubelet/pods/1a86ecbd-0a65-4736-9b70-669c92109124/volumes"
Oct 30 03:23:23.069641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.069624    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" path="/var/lib/kubelet/pods/2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0/volumes"
Oct 30 03:23:23.085385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.085366    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"
Oct 30 03:23:23.085516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.085398    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg" event={"ID":"fe390f2c-87b9-4f51-b0d6-59f185ac2c5f","Type":"ContainerDied","Data":"18966aea9564194fe09ce800eb0c5195beaf193b3ffb4b0b782fd2059d37accf"}
Oct 30 03:23:23.085516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.085444    4214 scope.go:122] "RemoveContainer" containerID="df8f8ceedfcffe7ec2a6856b00bfe8f5b9b0ff3833ded492814e5f38bbd4bfbb"
Oct 30 03:23:23.088361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.088339    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"
Oct 30 03:23:23.088481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.088357    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm" event={"ID":"1991f52c-7032-403a-8ab0-6e02949aa979","Type":"ContainerDied","Data":"666857447ba4c7c9b436acb65f3cade054ef8b41835a8c0d1f7fa71f29a3811c"}
Oct 30 03:23:23.097269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.097223    4214 scope.go:122] "RemoveContainer" containerID="62671c77e0ac7fea83876eb8aa8c70f84db039de44365e02eb51cbf215828e14"
Oct 30 03:23:23.145672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.145643    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:23:23.157872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.157013    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07e5c2-64b747b67-zh9zg"]
Oct 30 03:23:23.193508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.193481    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:23:23.197613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.197589    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07e5c2-86ddf96f78-mndhm"]
Oct 30 03:23:23.949488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:23.949446    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:24.866588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:24.866530    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07e5c2-7547dd4899-qp6fc" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.251:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:23:25.071878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:25.071840    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1991f52c-7032-403a-8ab0-6e02949aa979" path="/var/lib/kubelet/pods/1991f52c-7032-403a-8ab0-6e02949aa979/volumes"
Oct 30 03:23:25.072562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:25.072532    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" path="/var/lib/kubelet/pods/fe390f2c-87b9-4f51-b0d6-59f185ac2c5f/volumes"
Oct 30 03:23:29.117573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:29.117539    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:23:29.118094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:29.117733    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy" containerID="cri-o://659733fafe122146975dd4672d2da057a4ad0ad772f0ea450edd08c8a0adfde0" gracePeriod=30
Oct 30 03:23:29.665110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:29.665076    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:23:33.949225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:33.949186    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:34.328030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.327999    4214 generic.go:356] "Generic (PLEG): container finished" podID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerID="659733fafe122146975dd4672d2da057a4ad0ad772f0ea450edd08c8a0adfde0" exitCode=0
Oct 30 03:23:34.328176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.328072    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerDied","Data":"659733fafe122146975dd4672d2da057a4ad0ad772f0ea450edd08c8a0adfde0"}
Oct 30 03:23:34.328176 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.328136    4214 scope.go:122] "RemoveContainer" containerID="777084db5f9b9a18e15d87f6266b43aa27c334b666e9482e6766b0ddaa1e9097"
Oct 30 03:23:34.447291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.447214    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:23:34.576494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576466    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.576704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576557    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.576704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576594    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.576704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.576895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576727    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.576895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576754    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:34.576895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576852    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:34.577078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576905    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.577078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576940    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.577078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.576983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.577078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577033    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\") pod \"f51c6251-7198-4fba-be5c-e2504a0dee51\" (UID: \"f51c6251-7198-4fba-be5c-e2504a0dee51\") "
Oct 30 03:23:34.577078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577038    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:34.577365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577151    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:34.577427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577393    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:34.577803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577773    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.577908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577808    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51c6251-7198-4fba-be5c-e2504a0dee51-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.577908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577824    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.577908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577839    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.577908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.577853    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.579056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.579024    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:34.579268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.579222    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:34.579366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.579330    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "kube-api-access-gwlcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:34.579802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.579778    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo" pod "f51c6251-7198-4fba-be5c-e2504a0dee51" (UID: "f51c6251-7198-4fba-be5c-e2504a0dee51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:23:34.665399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.665358    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" podUID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" containerName="payload-processing" containerID="cri-o://570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b" gracePeriod=25
Oct 30 03:23:34.679453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.679426    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwlcx\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-kube-api-access-gwlcx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.679453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.679454    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.679597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.679463    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:34.679597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:34.679472    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51c6251-7198-4fba-be5c-e2504a0dee51-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:35.012149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.011816    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:23:35.084654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.084623    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\") pod \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") "
Oct 30 03:23:35.084808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.084703    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\" (UniqueName: \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\") pod \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\" (UID: \"36113c7c-5e08-46ca-af47-57bb5cd9ae9a\") "
Oct 30 03:23:35.085048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.085022    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume" pod "36113c7c-5e08-46ca-af47-57bb5cd9ae9a" (UID: "36113c7c-5e08-46ca-af47-57bb5cd9ae9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:35.085362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.085343    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:35.087170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.087148    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf" pod "36113c7c-5e08-46ca-af47-57bb5cd9ae9a" (UID: "36113c7c-5e08-46ca-af47-57bb5cd9ae9a"). InnerVolumeSpecName "kube-api-access-4xtqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:35.187501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.187459    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xtqf\" (UniqueName: \"kubernetes.io/projected/36113c7c-5e08-46ca-af47-57bb5cd9ae9a-kube-api-access-4xtqf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:35.338533 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.338503    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"
Oct 30 03:23:35.338725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.338547    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956" event={"ID":"f51c6251-7198-4fba-be5c-e2504a0dee51","Type":"ContainerDied","Data":"c3ebaf1b31ad25363dd9de38d99828058a4067dae10d7447ade2f02bc22c8df0"}
Oct 30 03:23:35.338725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.338607    4214 scope.go:122] "RemoveContainer" containerID="659733fafe122146975dd4672d2da057a4ad0ad772f0ea450edd08c8a0adfde0"
Oct 30 03:23:35.340484 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.340458    4214 generic.go:356] "Generic (PLEG): container finished" podID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" containerID="570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b" exitCode=0
Oct 30 03:23:35.340671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.340515    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"
Oct 30 03:23:35.340671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.340512    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" event={"ID":"36113c7c-5e08-46ca-af47-57bb5cd9ae9a","Type":"ContainerDied","Data":"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"}
Oct 30 03:23:35.340671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.340644    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q" event={"ID":"36113c7c-5e08-46ca-af47-57bb5cd9ae9a","Type":"ContainerDied","Data":"609bf34d8dfa171b7e232e649dcda89c222955b255684808a0c603377cef6c87"}
Oct 30 03:23:35.352354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.352326    4214 scope.go:122] "RemoveContainer" containerID="570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"
Oct 30 03:23:35.364050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.364031    4214 scope.go:122] "RemoveContainer" containerID="570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"
Oct 30 03:23:35.364362 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:23:35.364340    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b\": container with ID starting with 570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b not found: ID does not exist" containerID="570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"
Oct 30 03:23:35.364445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.364371    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b"} err="failed to get container status \"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b\": rpc error: code = NotFound desc = could not find container \"570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b\": container with ID starting with 570c3f448b31d7eede987a66d88300fc664d78f64cd8277aed098598edf4da3b not found: ID does not exist"
Oct 30 03:23:35.396820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.396763    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:23:35.409580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.409558    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-07e5c2-openshift-default-7789c64df5-r9956"]
Oct 30 03:23:35.439000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.438974    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:23:35.444020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:35.443996    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-5f8d6bf7f8-tsd7q"]
Oct 30 03:23:36.753220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.753181    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:23:36.754449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754427    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:36.754510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754451    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:36.754510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754500    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerName="main"
Oct 30 03:23:36.754510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754508    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerName="main"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754537    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" containerName="maas-api"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754545    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" containerName="maas-api"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754565    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerName="main"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754573    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerName="main"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754591    4214 cpu_manager.go:414] "containerMap: removing container" podUID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" containerName="payload-processing"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754598    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" containerName="payload-processing"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754608    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1991f52c-7032-403a-8ab0-6e02949aa979" containerName="payload-processing"
Oct 30 03:23:36.754625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754616    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1991f52c-7032-403a-8ab0-6e02949aa979" containerName="payload-processing"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754637    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" containerName="payload-pre-processing"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754645    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" containerName="payload-pre-processing"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754818    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754833    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef7941a-6f8b-45ec-b2cb-5e2b892e0164" containerName="main"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754848    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a86ecbd-0a65-4736-9b70-669c92109124" containerName="maas-api"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754858    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2141cb37-2374-4ba7-8fcb-58f9c0dcc8e0" containerName="main"
Oct 30 03:23:36.754879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754871    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" containerName="payload-processing"
Oct 30 03:23:36.755118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754881    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1991f52c-7032-403a-8ab0-6e02949aa979" containerName="payload-processing"
Oct 30 03:23:36.755118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754891    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe390f2c-87b9-4f51-b0d6-59f185ac2c5f" containerName="payload-pre-processing"
Oct 30 03:23:36.755118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.754901    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:36.759163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.759141    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.780182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.779819    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-4b5459-serving-cert\""
Oct 30 03:23:36.786288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.785063    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:23:36.809804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.809775    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.809957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.809828    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j627t\" (UniqueName: \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.810109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.810082    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.810324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.810301    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.911541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.911510    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.911746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.911724    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.911832 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.911813    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.911885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.911861    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j627t\" (UniqueName: \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.912602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.912574    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.914328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.914304    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.914500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.914481    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:36.931861 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:36.931840    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j627t\" (UniqueName: \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\") pod \"maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:37.087647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:37.084767    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:37.128296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:37.125931    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36113c7c-5e08-46ca-af47-57bb5cd9ae9a" path="/var/lib/kubelet/pods/36113c7c-5e08-46ca-af47-57bb5cd9ae9a/volumes"
Oct 30 03:23:37.128296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:37.126856    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" path="/var/lib/kubelet/pods/f51c6251-7198-4fba-be5c-e2504a0dee51/volumes"
Oct 30 03:23:37.436512 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:23:37.436481    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f45ba8_c10a_4a44_b87d_7f64ef6f1693.slice/crio-99ed7f3032dcc37d56f896893658ffeb4324a8b4e22235488a6773e05d9476d5 WatchSource:0}: Error finding container 99ed7f3032dcc37d56f896893658ffeb4324a8b4e22235488a6773e05d9476d5: Status 404 returned error can't find the container with id 99ed7f3032dcc37d56f896893658ffeb4324a8b4e22235488a6773e05d9476d5
Oct 30 03:23:37.437817 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:37.437790    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:23:38.378278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.378223    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" event={"ID":"66f45ba8-c10a-4a44-b87d-7f64ef6f1693","Type":"ContainerStarted","Data":"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"}
Oct 30 03:23:38.378690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.378297    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" event={"ID":"66f45ba8-c10a-4a44-b87d-7f64ef6f1693","Type":"ContainerStarted","Data":"99ed7f3032dcc37d56f896893658ffeb4324a8b4e22235488a6773e05d9476d5"}
Oct 30 03:23:38.378690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.378327    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:38.423103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.423046    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" podStartSLOduration=2.083117881 podStartE2EDuration="2.423032524s" totalImagesPullingTime="339.914643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:38.422388842 +0000 UTC m=+2557.496559635" watchObservedRunningTime="2026-10-30 03:23:38.423032524 +0000 UTC m=+2557.497203317"
Oct 30 03:23:38.749877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.749791    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:23:38.750663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.750640    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:38.750784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.750663    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51c6251-7198-4fba-be5c-e2504a0dee51" containerName="istio-proxy"
Oct 30 03:23:38.753728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.753706    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.772247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.772202    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-948965-serving-cert\""
Oct 30 03:23:38.780488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.780457    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:23:38.852766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.852733    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.852955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.852787    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hwx\" (UniqueName: \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.853053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.853031    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.853117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.853106    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.954525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.954488    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.954745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.954593    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.954745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.954655    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.954745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.954710    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hwx\" (UniqueName: \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.955553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.955525    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.957434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.957408    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.957575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.957550    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:38.984968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:38.984937    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hwx\" (UniqueName: \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\") pod \"maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:39.077253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:39.076528    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:39.193941 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:23:39.193884    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:23:39.400380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:39.400350    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:23:39.400706 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:23:39.400377    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7c1a97_08ef_4ea8_8d2b_8b8029ed7cfe.slice/crio-cde6301c654233eeae317ac5df8d0111a43b211759a3662272bcfeda153c110b WatchSource:0}: Error finding container cde6301c654233eeae317ac5df8d0111a43b211759a3662272bcfeda153c110b: Status 404 returned error can't find the container with id cde6301c654233eeae317ac5df8d0111a43b211759a3662272bcfeda153c110b
Oct 30 03:23:40.396526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:40.396488    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" event={"ID":"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe","Type":"ContainerStarted","Data":"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"}
Oct 30 03:23:40.396704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:40.396554    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:40.396704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:40.396570    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" event={"ID":"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe","Type":"ContainerStarted","Data":"cde6301c654233eeae317ac5df8d0111a43b211759a3662272bcfeda153c110b"}
Oct 30 03:23:40.454350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:40.454215    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" podStartSLOduration=2.200857486 podStartE2EDuration="2.454200905s" totalImagesPullingTime="253.343419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:40.452309471 +0000 UTC m=+2559.526480269" watchObservedRunningTime="2026-10-30 03:23:40.454200905 +0000 UTC m=+2559.528371699"
Oct 30 03:23:42.418066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.418033    4214 generic.go:356] "Generic (PLEG): container finished" podID="c8480052-5a01-473f-bc4a-45a3472e434e" containerID="ec16579a17b96b19147285f345e7d1d97e610e3a4ca18cbd1369a2127509475e" exitCode=0
Oct 30 03:23:42.418511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.418351    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" event={"ID":"c8480052-5a01-473f-bc4a-45a3472e434e","Type":"ContainerDied","Data":"ec16579a17b96b19147285f345e7d1d97e610e3a4ca18cbd1369a2127509475e"}
Oct 30 03:23:42.508903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.508876    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:23:42.668253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.668054    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\" (UniqueName: \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\") pod \"c8480052-5a01-473f-bc4a-45a3472e434e\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") "
Oct 30 03:23:42.668253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.668164    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\") pod \"c8480052-5a01-473f-bc4a-45a3472e434e\" (UID: \"c8480052-5a01-473f-bc4a-45a3472e434e\") "
Oct 30 03:23:42.686295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.668760    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume" pod "c8480052-5a01-473f-bc4a-45a3472e434e" (UID: "c8480052-5a01-473f-bc4a-45a3472e434e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:42.705742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.702585    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74" pod "c8480052-5a01-473f-bc4a-45a3472e434e" (UID: "c8480052-5a01-473f-bc4a-45a3472e434e"). InnerVolumeSpecName "kube-api-access-l6s74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:42.769888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.769809    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6s74\" (UniqueName: \"kubernetes.io/projected/c8480052-5a01-473f-bc4a-45a3472e434e-kube-api-access-l6s74\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:42.769888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:42.769841    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8480052-5a01-473f-bc4a-45a3472e434e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:23:43.426980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:43.426902    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"
Oct 30 03:23:43.426980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:43.426935    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt" event={"ID":"c8480052-5a01-473f-bc4a-45a3472e434e","Type":"ContainerDied","Data":"e45148a48d268d64969691001ce4ee71bdb5cdc0d7b53a95b4ff24abd0cdc9ed"}
Oct 30 03:23:43.427487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:43.427000    4214 scope.go:122] "RemoveContainer" containerID="ec16579a17b96b19147285f345e7d1d97e610e3a4ca18cbd1369a2127509475e"
Oct 30 03:23:43.505843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:43.505808    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:23:43.518305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:43.518275    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9245d1-69495f589f-cv5xt"]
Oct 30 03:23:44.393207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:44.393175    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:23:45.069792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:45.069754    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" path="/var/lib/kubelet/pods/c8480052-5a01-473f-bc4a-45a3472e434e/volumes"
Oct 30 03:23:46.410528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:46.410496    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:23:49.287559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.287518    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:23:49.288359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.288338    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing"
Oct 30 03:23:49.288359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.288359    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing"
Oct 30 03:23:49.288596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.288580    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8480052-5a01-473f-bc4a-45a3472e434e" containerName="payload-pre-processing"
Oct 30 03:23:49.293114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.293091    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.327988 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.327952    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:23:49.342481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.342451    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-d823d1\"/\"auth-test-model-d823d1-kserve-self-signed-certs\""
Oct 30 03:23:49.342678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.342464    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-d823d1\"/\"openshift-service-ca.crt\""
Oct 30 03:23:49.342779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.342498    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-d823d1\"/\"default-dockercfg-rmrnf\""
Oct 30 03:23:49.342779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.342564    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-d823d1\"/\"kube-root-ca.crt\""
Oct 30 03:23:49.460125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460095    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzz76\" (UniqueName: \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.460315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460177    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.460435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460410    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.460497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460478    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.460551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460532    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.460589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.460565    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.567624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.567576    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.568415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.568393    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.569129 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.569108    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.569312 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.569294    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.569552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.569532    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz76\" (UniqueName: \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.579270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.569746    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.580015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.572559    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.580215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.568306    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.580351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.572775    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.602219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.602150    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzz76\" (UniqueName: \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.622001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.619047    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.622001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.621947    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\") pod \"auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") " pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:49.914816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:49.914738    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:50.233270 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:23:50.233215    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d6374ea_3bbe_4b1b_bedb_9cdd397e3123.slice/crio-182f013bd029aed0a66a90d1a0a24b643fb0db66522d75631cb1339d8cc66ebf WatchSource:0}: Error finding container 182f013bd029aed0a66a90d1a0a24b643fb0db66522d75631cb1339d8cc66ebf: Status 404 returned error can't find the container with id 182f013bd029aed0a66a90d1a0a24b643fb0db66522d75631cb1339d8cc66ebf
Oct 30 03:23:50.233270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:50.233252    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:23:50.498858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:50.498750    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" event={"ID":"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123","Type":"ContainerStarted","Data":"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"}
Oct 30 03:23:50.498858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:50.498827    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" event={"ID":"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123","Type":"ContainerStarted","Data":"182f013bd029aed0a66a90d1a0a24b643fb0db66522d75631cb1339d8cc66ebf"}
Oct 30 03:23:50.552561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:50.552514    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" podStartSLOduration=1.552500834 podStartE2EDuration="1.552500834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:50.549219572 +0000 UTC m=+2569.623390364" watchObservedRunningTime="2026-10-30 03:23:50.552500834 +0000 UTC m=+2569.626671652"
Oct 30 03:23:50.915218 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:50.915158    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:59.915095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:59.915058    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:23:59.928638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:23:59.928570    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:24:00.670076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:00.670012    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:24:01.663823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.663731    4214 generic.go:356] "Generic (PLEG): container finished" podID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerID="0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3" exitCode=0
Oct 30 03:24:01.663823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.663802    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerDied","Data":"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"}
Oct 30 03:24:01.664638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.664544    4214 scope.go:122] "RemoveContainer" containerID="0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"
Oct 30 03:24:01.666653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.666334    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:01.666653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.666424    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:01.666653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:01.666471    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:02.306765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.306726    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:24:02.307001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.306976    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy" containerID="cri-o://8c017575d9c3bc2b1a309bc6124c8080e8858334aa11a3d02229b304a38c2133" gracePeriod=30
Oct 30 03:24:02.700870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.700389    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerStarted","Data":"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"}
Oct 30 03:24:02.819031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.818993    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:24:02.820117 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.820090    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:02.820261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:02.820139    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:03.819596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:03.819558    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:03.820164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:03.819622    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:04.819595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:04.819556    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:04.819800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:04.819614    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:05.818789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:05.818752    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:05.818991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:05.818808    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:06.819670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:06.819635    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:06.820075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:06.819699    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:07.255432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.255337    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:24:07.829636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.829597    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 30 03:24:07.830157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.830115    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 30 03:24:07.830713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.830512    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 03:24:07.879374 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.879336    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 03:24:07.891416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.891350    4214 generic.go:356] "Generic (PLEG): container finished" podID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerID="8c017575d9c3bc2b1a309bc6124c8080e8858334aa11a3d02229b304a38c2133" exitCode=0
Oct 30 03:24:07.891766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.891740    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerDied","Data":"8c017575d9c3bc2b1a309bc6124c8080e8858334aa11a3d02229b304a38c2133"}
Oct 30 03:24:07.891975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.891959    4214 scope.go:122] "RemoveContainer" containerID="e0fd857a00ec941296244fb4e4886bcaeb3445c75bc9cebd48cc6a58aebb757e"
Oct 30 03:24:07.952337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.941312    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:24:07.971729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:07.971697    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.020259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.014980    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a0f7e\""
Oct 30 03:24:08.048385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.017219    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a0f7e-dockercfg-ncnr4\""
Oct 30 03:24:08.098407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.098310    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtgt\" (UniqueName: \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.098932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.098902    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.103512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.103488    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:24:08.232412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.232373    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtgt\" (UniqueName: \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.233573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.233531    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.265256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.242667    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.316573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.316508    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtgt\" (UniqueName: \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.415321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.413005    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:24:08.415321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.413271    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy" containerID="cri-o://e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1" gracePeriod=30
Oct 30 03:24:08.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.425095    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:24:08.429253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.425882    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy" containerID="cri-o://a6f4689a226495600663cb9dbbb26bb9ab3afc8a40f6518c21af80ac6118f7b5" gracePeriod=30
Oct 30 03:24:08.440967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.440709    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:24:08.440967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.440958    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy" containerID="cri-o://474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9" gracePeriod=30
Oct 30 03:24:08.445030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.444996    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:24:08.445507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.445433    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy" containerID="cri-o://71dc7a8f99ce9271f68db837da20ffd42d9bd60dd065ac43d9df49b71d275b5b" gracePeriod=30
Oct 30 03:24:08.700938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.602925    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:08.726760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.725215    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795086    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795262    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795327    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795364    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795428    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795454    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795579    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795639    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.800270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.795695    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\") pod \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\" (UID: \"80d38247-1126-46aa-9bc0-d15ac86cb2d1\") "
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.802106    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.802437    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.803047    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.803195    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.804303    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.804368    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.804731    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:08.807386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.807221    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:08.813455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.809464    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb" pod "80d38247-1126-46aa-9bc0-d15ac86cb2d1" (UID: "80d38247-1126-46aa-9bc0-d15ac86cb2d1"). InnerVolumeSpecName "kube-api-access-5vvlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:08.900223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900192    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900222    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900250    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900264    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvlb\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-kube-api-access-5vvlb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900277    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900289    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900299    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d38247-1126-46aa-9bc0-d15ac86cb2d1-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900311    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.900767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.900324    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d38247-1126-46aa-9bc0-d15ac86cb2d1-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:08.903330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.903305    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"
Oct 30 03:24:08.903459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.903331    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx" event={"ID":"80d38247-1126-46aa-9bc0-d15ac86cb2d1","Type":"ContainerDied","Data":"3ed198ae99e008421af20020f2fa205ced4b2512907e5995c7e9354f16ccb481"}
Oct 30 03:24:08.903459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.903400    4214 scope.go:122] "RemoveContainer" containerID="8c017575d9c3bc2b1a309bc6124c8080e8858334aa11a3d02229b304a38c2133"
Oct 30 03:24:08.958913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.958815    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:24:08.961543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.961463    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:24:08.973283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.973242    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:24:08.979580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:08.979546    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-734274-openshift-default-5ddb7bbcbc-n6wwx"]
Oct 30 03:24:09.050783 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:09.050381    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19faf67_97c9_49d4_a80b_a32dc8c42204.slice/crio-12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c WatchSource:0}: Error finding container 12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c: Status 404 returned error can't find the container with id 12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c
Oct 30 03:24:09.058690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.058643    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:24:09.094038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.093544    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" path="/var/lib/kubelet/pods/80d38247-1126-46aa-9bc0-d15ac86cb2d1/volumes"
Oct 30 03:24:09.222360 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:09.222326    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:24:09.914408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.914370    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" event={"ID":"e19faf67-97c9-49d4-a80b-a32dc8c42204","Type":"ContainerStarted","Data":"e864afa9e2f9e9cb9eb805e30e30e6f6c4bdd7f43d4ceea47480b5b059babeb5"}
Oct 30 03:24:09.914930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.914455    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" event={"ID":"e19faf67-97c9-49d4-a80b-a32dc8c42204","Type":"ContainerStarted","Data":"12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c"}
Oct 30 03:24:09.914930 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.914486    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:09.961249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:09.960939    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" podStartSLOduration=2.96091734 podStartE2EDuration="2.96091734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:09.957514938 +0000 UTC m=+2589.031685730" watchObservedRunningTime="2026-10-30 03:24:09.96091734 +0000 UTC m=+2589.035088147"
Oct 30 03:24:13.428371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.428334    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 03:24:13.874513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.874315    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:24:13.950666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.942974    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:24:13.987163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.986483    4214 generic.go:356] "Generic (PLEG): container finished" podID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerID="474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9" exitCode=0
Oct 30 03:24:13.987163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.986644    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"
Oct 30 03:24:13.987163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.986810    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" event={"ID":"b74f76ad-f1e6-406e-add6-14a7f48429bb","Type":"ContainerDied","Data":"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"}
Oct 30 03:24:13.987163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.986851    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq" event={"ID":"b74f76ad-f1e6-406e-add6-14a7f48429bb","Type":"ContainerDied","Data":"064d04bb37b6df1cf2348cbf4946c851df25085c23c03429ac5e161f8147cf6f"}
Oct 30 03:24:13.987163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:13.986881    4214 scope.go:122] "RemoveContainer" containerID="474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"
Oct 30 03:24:14.005769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.005725    4214 generic.go:356] "Generic (PLEG): container finished" podID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerID="a6f4689a226495600663cb9dbbb26bb9ab3afc8a40f6518c21af80ac6118f7b5" exitCode=0
Oct 30 03:24:14.006051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.006024    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" event={"ID":"abbd3dcf-9b5b-431d-84b2-aa207d044e74","Type":"ContainerDied","Data":"a6f4689a226495600663cb9dbbb26bb9ab3afc8a40f6518c21af80ac6118f7b5"}
Oct 30 03:24:14.009444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.009416    4214 generic.go:356] "Generic (PLEG): container finished" podID="91188950-a29c-4667-86a5-ed130774a70d" containerID="71dc7a8f99ce9271f68db837da20ffd42d9bd60dd065ac43d9df49b71d275b5b" exitCode=0
Oct 30 03:24:14.009590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.009527    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" event={"ID":"91188950-a29c-4667-86a5-ed130774a70d","Type":"ContainerDied","Data":"71dc7a8f99ce9271f68db837da20ffd42d9bd60dd065ac43d9df49b71d275b5b"}
Oct 30 03:24:14.017669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.017640    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.017866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.017850    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.017978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.017888    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.017978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.017926    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.017978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.017968    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021225    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021304    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021353    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021474    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021575 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021546    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021583    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021635    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021690    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021742    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021773    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021809    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\") pod \"b74f76ad-f1e6-406e-add6-14a7f48429bb\" (UID: \"b74f76ad-f1e6-406e-add6-14a7f48429bb\") "
Oct 30 03:24:14.021976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021879    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.021976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.021932    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\") pod \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\" (UID: \"384248e9-5440-4b3e-851f-4b3a2dcd9f7c\") "
Oct 30 03:24:14.023244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.023207    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:14.023641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.023624    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.024371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.024350    4214 generic.go:356] "Generic (PLEG): container finished" podID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerID="e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1" exitCode=0
Oct 30 03:24:14.027133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.026437    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "kube-api-access-mk6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.027283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.026925    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.027452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.027440    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"
Oct 30 03:24:14.027981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.027955    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerDied","Data":"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"}
Oct 30 03:24:14.028074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.028007    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl" event={"ID":"384248e9-5440-4b3e-851f-4b3a2dcd9f7c","Type":"ContainerDied","Data":"046efcc684d85521f55369373a1061221cd1456b118b0e6139a2815e05deb962"}
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.028884    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.029441    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.029524    4214 scope.go:122] "RemoveContainer" containerID="474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.029673    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:14.029859    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9\": container with ID starting with 474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9 not found: ID does not exist" containerID="474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.029890    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9"} err="failed to get container status \"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9\": rpc error: code = NotFound desc = could not find container \"474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9\": container with ID starting with 474acaa5af55a8c8350fdef29465f6991d034f52eb300430910d836278b913d9 not found: ID does not exist"
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.029913    4214 scope.go:122] "RemoveContainer" containerID="e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.030002    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.030011    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:14.031445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.030331    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.033052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.033023    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.033353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.033329    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.034392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.034219    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:14.036177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.036143    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "kube-api-access-99cxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.036204    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.036301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.036214    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token" pod "384248e9-5440-4b3e-851f-4b3a2dcd9f7c" (UID: "384248e9-5440-4b3e-851f-4b3a2dcd9f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.036431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.036399    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.037399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.037380    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo" pod "b74f76ad-f1e6-406e-add6-14a7f48429bb" (UID: "b74f76ad-f1e6-406e-add6-14a7f48429bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:14.044888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.044870    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:24:14.046427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.046413    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:24:14.047114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.047099    4214 scope.go:122] "RemoveContainer" containerID="017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"
Oct 30 03:24:14.058847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.058825    4214 scope.go:122] "RemoveContainer" containerID="e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"
Oct 30 03:24:14.059190 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:14.059167    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1\": container with ID starting with e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1 not found: ID does not exist" containerID="e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"
Oct 30 03:24:14.059310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.059200    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1"} err="failed to get container status \"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1\": rpc error: code = NotFound desc = could not find container \"e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1\": container with ID starting with e33a249959886675c32bf6a329cd23bf3e4b7181af4b5095b18d5d81b48c49b1 not found: ID does not exist"
Oct 30 03:24:14.059310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.059224    4214 scope.go:122] "RemoveContainer" containerID="017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"
Oct 30 03:24:14.059592 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:14.059569    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658\": container with ID starting with 017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658 not found: ID does not exist" containerID="017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"
Oct 30 03:24:14.059660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.059603    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658"} err="failed to get container status \"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658\": rpc error: code = NotFound desc = could not find container \"017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658\": container with ID starting with 017d6f9e3a92b43a5af902884c6040470e52114404b9789a71b2a6eb17d66658 not found: ID does not exist"
Oct 30 03:24:14.132907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.132869    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.133245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.133208    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.133421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.133404    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.137366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.137337    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.137548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.137527    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.144248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.137678    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.144506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.144485    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.144635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.144621    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.144778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.144763    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.182253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.144913    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.182678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.182625    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.182860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.182842    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.183051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.183032    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.183220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.183205    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.183660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.183638    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\") pod \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\" (UID: \"abbd3dcf-9b5b-431d-84b2-aa207d044e74\") "
Oct 30 03:24:14.191393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.191319    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.195082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.191571    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.146636    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.147507    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.147686    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.152310    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.161413    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.172882    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.173682    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.178392    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.190821    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.196582    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.197039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.196820    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.205845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.205784    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\") pod \"91188950-a29c-4667-86a5-ed130774a70d\" (UID: \"91188950-a29c-4667-86a5-ed130774a70d\") "
Oct 30 03:24:14.206925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.206902    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207093 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207076    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207188    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207338    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91188950-a29c-4667-86a5-ed130774a70d-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207430    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207540    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207626    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207727    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207828    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.207919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207910    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208005 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.207994    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99cxm\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-kube-api-access-99cxm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208081    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208163    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208291 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208278    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208377    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208460    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208539    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74f76ad-f1e6-406e-add6-14a7f48429bb-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208627    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-kube-api-access-mk6db\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208718    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208803    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208885    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.208982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.208971    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209058    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209139    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209248    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209353    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74f76ad-f1e6-406e-add6-14a7f48429bb-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209464    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209556    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209641    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384248e9-5440-4b3e-851f-4b3a2dcd9f7c-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.209939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.209920    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.218848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.218797    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "kube-api-access-6gsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.219269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.219105    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:14.221367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.221344    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:14.232422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.232359    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.269347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.269302    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy" pod "91188950-a29c-4667-86a5-ed130774a70d" (UID: "91188950-a29c-4667-86a5-ed130774a70d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:14.270711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.270667    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr" pod "abbd3dcf-9b5b-431d-84b2-aa207d044e74" (UID: "abbd3dcf-9b5b-431d-84b2-aa207d044e74"). InnerVolumeSpecName "kube-api-access-fnxfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:14.311514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311488    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311515    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91188950-a29c-4667-86a5-ed130774a70d-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311524    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxfr\" (UniqueName: \"kubernetes.io/projected/abbd3dcf-9b5b-431d-84b2-aa207d044e74-kube-api-access-fnxfr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311534    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311543    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd3dcf-9b5b-431d-84b2-aa207d044e74-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311551    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsdb\" (UniqueName: \"kubernetes.io/projected/91188950-a29c-4667-86a5-ed130774a70d-kube-api-access-6gsdb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.311674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.311561    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91188950-a29c-4667-86a5-ed130774a70d-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.348025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.347999    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:24:14.352833 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.352811    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-p46sq"]
Oct 30 03:24:14.386999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.386964    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:24:14.394133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:14.394109    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-lfvgl"]
Oct 30 03:24:15.039084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.039048    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"
Oct 30 03:24:15.039084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.039063    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf" event={"ID":"91188950-a29c-4667-86a5-ed130774a70d","Type":"ContainerDied","Data":"ded805fad56fba49e3609df741f34ea2eab92a3ca977fef3b388ab91f061797c"}
Oct 30 03:24:15.039678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.039121    4214 scope.go:122] "RemoveContainer" containerID="71dc7a8f99ce9271f68db837da20ffd42d9bd60dd065ac43d9df49b71d275b5b"
Oct 30 03:24:15.043826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.043620    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"
Oct 30 03:24:15.043826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.043645    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh" event={"ID":"abbd3dcf-9b5b-431d-84b2-aa207d044e74","Type":"ContainerDied","Data":"af63ffa7d0e1ae6e29d8a423b85901d1dcb95b0cac9cdd0e20accc247a360caf"}
Oct 30 03:24:15.053116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.053093    4214 scope.go:122] "RemoveContainer" containerID="a6f4689a226495600663cb9dbbb26bb9ab3afc8a40f6518c21af80ac6118f7b5"
Oct 30 03:24:15.080874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.080842    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" path="/var/lib/kubelet/pods/384248e9-5440-4b3e-851f-4b3a2dcd9f7c/volumes"
Oct 30 03:24:15.081465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.081441    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" path="/var/lib/kubelet/pods/b74f76ad-f1e6-406e-add6-14a7f48429bb/volumes"
Oct 30 03:24:15.105563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.105536    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:24:15.118750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.118721    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-8qsrf"]
Oct 30 03:24:15.149500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.149431    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:24:15.153609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:15.153578    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-sdrwh"]
Oct 30 03:24:17.069427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:17.069396    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91188950-a29c-4667-86a5-ed130774a70d" path="/var/lib/kubelet/pods/91188950-a29c-4667-86a5-ed130774a70d/volumes"
Oct 30 03:24:17.069908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:17.069893    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" path="/var/lib/kubelet/pods/abbd3dcf-9b5b-431d-84b2-aa207d044e74/volumes"
Oct 30 03:24:18.429671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:18.429629    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing" containerID="cri-o://97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80" gracePeriod=25
Oct 30 03:24:18.530674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:18.530625    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:20.928451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:20.928418    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:24:23.143459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.143306    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" containerID="cri-o://c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71" gracePeriod=45
Oct 30 03:24:23.178948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.178906    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main" containerID="cri-o://d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58" gracePeriod=45
Oct 30 03:24:23.589867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.589831    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:24:23.594512 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.594482    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:24:23.649216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649172    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.649493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649283    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649393    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649434    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.649493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649485    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649614    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.649718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649658    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649761    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649796    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\" (UniqueName: \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.649931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649826    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\") pod \"44790c98-f4de-4749-bc91-6170be8351fc\" (UID: \"44790c98-f4de-4749-bc91-6170be8351fc\") "
Oct 30 03:24:23.649931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649872    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.649931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.649905    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\") pod \"04c0f412-50d8-4e14-9c47-4275c9def39e\" (UID: \"04c0f412-50d8-4e14-9c47-4275c9def39e\") "
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.650591    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.650845    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.651045    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.651104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.651168    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.652940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.652524    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.656410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.655042    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "kube-api-access-6h4kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:23.662586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.662545    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "kube-api-access-x2kkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:23.666672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.666631    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.668334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.668262    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:23.671644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.671602    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm" pod "04c0f412-50d8-4e14-9c47-4275c9def39e" (UID: "04c0f412-50d8-4e14-9c47-4275c9def39e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.673499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.673466    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs" pod "44790c98-f4de-4749-bc91-6170be8351fc" (UID: "44790c98-f4de-4749-bc91-6170be8351fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:23.752586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752530    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752576    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kkr\" (UniqueName: \"kubernetes.io/projected/44790c98-f4de-4749-bc91-6170be8351fc-kube-api-access-x2kkr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752595    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752608    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752619    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0f412-50d8-4e14-9c47-4275c9def39e-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752629    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44790c98-f4de-4749-bc91-6170be8351fc-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752638    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752650    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4kk\" (UniqueName: \"kubernetes.io/projected/04c0f412-50d8-4e14-9c47-4275c9def39e-kube-api-access-6h4kk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752661    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44790c98-f4de-4749-bc91-6170be8351fc-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752672    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752683    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.752859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:23.752693    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04c0f412-50d8-4e14-9c47-4275c9def39e-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:24.217993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.217957    4214 generic.go:356] "Generic (PLEG): container finished" podID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerID="d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58" exitCode=0
Oct 30 03:24:24.218546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.218011    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"
Oct 30 03:24:24.218546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.218039    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" event={"ID":"04c0f412-50d8-4e14-9c47-4275c9def39e","Type":"ContainerDied","Data":"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"}
Oct 30 03:24:24.218546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.218092    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx" event={"ID":"04c0f412-50d8-4e14-9c47-4275c9def39e","Type":"ContainerDied","Data":"f4bde61434a0a2c958c83f8fdd944f185e00df12f3b2092c6dee780f8413280d"}
Oct 30 03:24:24.218546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.218116    4214 scope.go:122] "RemoveContainer" containerID="d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"
Oct 30 03:24:24.219803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.219783    4214 generic.go:356] "Generic (PLEG): container finished" podID="44790c98-f4de-4749-bc91-6170be8351fc" containerID="c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71" exitCode=0
Oct 30 03:24:24.219887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.219812    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" event={"ID":"44790c98-f4de-4749-bc91-6170be8351fc","Type":"ContainerDied","Data":"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"}
Oct 30 03:24:24.219887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.219835    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" event={"ID":"44790c98-f4de-4749-bc91-6170be8351fc","Type":"ContainerDied","Data":"9d3551b66af3f5638a55532a5bb78519ab32a65d7a3648f932c654e0b8bec2b3"}
Oct 30 03:24:24.219887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.219835    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"
Oct 30 03:24:24.230786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.230765    4214 scope.go:122] "RemoveContainer" containerID="d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"
Oct 30 03:24:24.231077 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:24.231060    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58\": container with ID starting with d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58 not found: ID does not exist" containerID="d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"
Oct 30 03:24:24.231131 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.231085    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58"} err="failed to get container status \"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58\": rpc error: code = NotFound desc = could not find container \"d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58\": container with ID starting with d242c8d04757b61c6e7e2ee782fb57d9a8b968a196ca8ee2b1e58c636bb23a58 not found: ID does not exist"
Oct 30 03:24:24.231131 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.231102    4214 scope.go:122] "RemoveContainer" containerID="c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"
Oct 30 03:24:24.244083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.244053    4214 scope.go:122] "RemoveContainer" containerID="c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"
Oct 30 03:24:24.244440 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:24.244415    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71\": container with ID starting with c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71 not found: ID does not exist" containerID="c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"
Oct 30 03:24:24.244532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.244450    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71"} err="failed to get container status \"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71\": rpc error: code = NotFound desc = could not find container \"c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71\": container with ID starting with c0247647cc2dc0d64dddf2ed842eb52fecc9e3cf680173167071cdf01eb0af71 not found: ID does not exist"
Oct 30 03:24:24.296396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.296364    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 03:24:24.303450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.303419    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-w2-810119-kserve-68bf5b7577-xwfcx"]
Oct 30 03:24:24.356660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.356617    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 03:24:24.377637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.377594    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x"]
Oct 30 03:24:24.493173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:24.493075    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/distinct-2-w2-810119-kserve-6777748b54-cjq8x" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:24:25.111797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:25.111758    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" path="/var/lib/kubelet/pods/04c0f412-50d8-4e14-9c47-4275c9def39e/volumes"
Oct 30 03:24:25.112795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:25.112767    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44790c98-f4de-4749-bc91-6170be8351fc" path="/var/lib/kubelet/pods/44790c98-f4de-4749-bc91-6170be8351fc/volumes"
Oct 30 03:24:28.530772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:28.530731    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:29.624022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.623980    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:24:29.624974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.624954    4214 cpu_manager.go:414] "containerMap: removing container" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.624976    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.624986    4214 cpu_manager.go:414] "containerMap: removing container" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.624991    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.624998    4214 cpu_manager.go:414] "containerMap: removing container" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625003    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625011    4214 cpu_manager.go:414] "containerMap: removing container" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625015    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625024    4214 cpu_manager.go:414] "containerMap: removing container" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625029    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625047    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy"
Oct 30 03:24:29.625052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625054    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625069    4214 cpu_manager.go:414] "containerMap: removing container" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625074    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625085    4214 cpu_manager.go:414] "containerMap: removing container" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625089    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625114    4214 cpu_manager.go:414] "containerMap: removing container" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625118    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625267    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c0f412-50d8-4e14-9c47-4275c9def39e" containerName="main"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625281    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625289    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="44790c98-f4de-4749-bc91-6170be8351fc" containerName="main"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625297    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="91188950-a29c-4667-86a5-ed130774a70d" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625303    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625311    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74f76ad-f1e6-406e-add6-14a7f48429bb" containerName="istio-proxy"
Oct 30 03:24:29.625551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.625318    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd3dcf-9b5b-431d-84b2-aa207d044e74" containerName="istio-proxy"
Oct 30 03:24:29.629581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.629560    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.650276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.650241    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:24:29.703309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.703272    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-8b0787-openshift-default-dockercfg-zdqbc\""
Oct 30 03:24:29.776091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.776043    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.776490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.776463    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.776652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.776636    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.776775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.776761    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.776944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.776929    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.777038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.777024    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.777295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.777277    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skks4\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.787263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.784306    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.787263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.784417    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.891443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.891351    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.892116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.892090    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.892862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.892842    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.893406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.893387    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.898362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.898334    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.899026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.898998    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.899557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.899538    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.903494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.903469    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skks4\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.904020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.903998    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.900505    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.903209    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.893304    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.899430    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.904487    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907772 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.898838    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.907853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.905334    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.929275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.928088    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.929275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.928222    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skks4\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\") pod \"e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") " pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:29.945071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:29.944785    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:30.326720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:30.326689    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:24:30.328047 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:30.328020    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7454b33a_0272_4cad_a8f2_8d2b1a939917.slice/crio-ace11b46d07b8168d9ed03cda66170f2e67692423e5a7ef1288063f98aa844a2 WatchSource:0}: Error finding container ace11b46d07b8168d9ed03cda66170f2e67692423e5a7ef1288063f98aa844a2: Status 404 returned error can't find the container with id ace11b46d07b8168d9ed03cda66170f2e67692423e5a7ef1288063f98aa844a2
Oct 30 03:24:30.330949 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:30.330919    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:30.331039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:30.330986    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:30.331039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:30.331028    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:31.288585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:31.288541    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerStarted","Data":"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"}
Oct 30 03:24:31.289067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:31.288597    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerStarted","Data":"ace11b46d07b8168d9ed03cda66170f2e67692423e5a7ef1288063f98aa844a2"}
Oct 30 03:24:31.344248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:31.344186    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podStartSLOduration=2.3441716980000002 podStartE2EDuration="2.344171698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:31.341578261 +0000 UTC m=+2610.415749055" watchObservedRunningTime="2026-10-30 03:24:31.344171698 +0000 UTC m=+2610.418342491"
Oct 30 03:24:31.955250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:31.950829    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:32.077400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:32.077365    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:32.296554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:32.296524    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:32.297700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:32.297676    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:24:33.392358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.392322    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:24:33.392850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.392610    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" podUID="c7622710-e54b-44c0-befb-12a7f29bedde" containerName="maas-api" containerID="cri-o://8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d" gracePeriod=30
Oct 30 03:24:33.817660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.817629    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:24:33.913916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.913877    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\") pod \"c7622710-e54b-44c0-befb-12a7f29bedde\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") "
Oct 30 03:24:33.914170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.914065    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\") pod \"c7622710-e54b-44c0-befb-12a7f29bedde\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") "
Oct 30 03:24:33.914356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.914254    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\") pod \"c7622710-e54b-44c0-befb-12a7f29bedde\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") "
Oct 30 03:24:33.914356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.914324    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\" (UniqueName: \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\") pod \"c7622710-e54b-44c0-befb-12a7f29bedde\" (UID: \"c7622710-e54b-44c0-befb-12a7f29bedde\") "
Oct 30 03:24:33.915066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.914818    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca" pod "c7622710-e54b-44c0-befb-12a7f29bedde" (UID: "c7622710-e54b-44c0-befb-12a7f29bedde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:33.915066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.914984    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7622710-e54b-44c0-befb-12a7f29bedde-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:33.917413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.917374    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7" pod "c7622710-e54b-44c0-befb-12a7f29bedde" (UID: "c7622710-e54b-44c0-befb-12a7f29bedde"). InnerVolumeSpecName "kube-api-access-jw6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:33.917562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.917535    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls" pod "c7622710-e54b-44c0-befb-12a7f29bedde" (UID: "c7622710-e54b-44c0-befb-12a7f29bedde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:33.917755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:33.917730    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs" pod "c7622710-e54b-44c0-befb-12a7f29bedde" (UID: "c7622710-e54b-44c0-befb-12a7f29bedde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:34.019884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.019801    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:34.019884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.019841    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7622710-e54b-44c0-befb-12a7f29bedde-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:34.019884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.019856    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6p7\" (UniqueName: \"kubernetes.io/projected/c7622710-e54b-44c0-befb-12a7f29bedde-kube-api-access-jw6p7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:34.314099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.314062    4214 generic.go:356] "Generic (PLEG): container finished" podID="c7622710-e54b-44c0-befb-12a7f29bedde" containerID="8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d" exitCode=0
Oct 30 03:24:34.314422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.314109    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"
Oct 30 03:24:34.314422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.314146    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" event={"ID":"c7622710-e54b-44c0-befb-12a7f29bedde","Type":"ContainerDied","Data":"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"}
Oct 30 03:24:34.314422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.314204    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5" event={"ID":"c7622710-e54b-44c0-befb-12a7f29bedde","Type":"ContainerDied","Data":"efeafb7653a01475427145160080a97fe51a9906670ada75b240de8c3fa86a9a"}
Oct 30 03:24:34.314422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.314244    4214 scope.go:122] "RemoveContainer" containerID="8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"
Oct 30 03:24:34.326542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.326517    4214 scope.go:122] "RemoveContainer" containerID="8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"
Oct 30 03:24:34.326847 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:34.326828    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d\": container with ID starting with 8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d not found: ID does not exist" containerID="8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"
Oct 30 03:24:34.326898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.326857    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d"} err="failed to get container status \"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d\": rpc error: code = NotFound desc = could not find container \"8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d\": container with ID starting with 8e1b1985e3ed34d6059bc1d27da0cc0c4a995ba147379711dba545f8c678dd7d not found: ID does not exist"
Oct 30 03:24:34.409748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.409707    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:24:34.424459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:34.424427    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-360818-64b99dd876-lzds5"]
Oct 30 03:24:35.068647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:35.068612    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7622710-e54b-44c0-befb-12a7f29bedde" path="/var/lib/kubelet/pods/c7622710-e54b-44c0-befb-12a7f29bedde/volumes"
Oct 30 03:24:36.725952 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.725915    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:24:36.726748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.726730    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c7622710-e54b-44c0-befb-12a7f29bedde" containerName="maas-api"
Oct 30 03:24:36.726795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.726750    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7622710-e54b-44c0-befb-12a7f29bedde" containerName="maas-api"
Oct 30 03:24:36.726872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.726863    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d38247-1126-46aa-9bc0-d15ac86cb2d1" containerName="istio-proxy"
Oct 30 03:24:36.726913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.726876    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7622710-e54b-44c0-befb-12a7f29bedde" containerName="maas-api"
Oct 30 03:24:36.726913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.726884    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="384248e9-5440-4b3e-851f-4b3a2dcd9f7c" containerName="istio-proxy"
Oct 30 03:24:36.731809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.731791    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:36.755285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.754964    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-8b0787\""
Oct 30 03:24:36.763515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.763486    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-8b0787-dockercfg-9mp6j\""
Oct 30 03:24:36.771210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.771177    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:24:36.861065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.861012    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:24:36.871305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.871262    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hgn\" (UniqueName: \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:36.905250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.901820    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:36.905250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.901571    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:36.965123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:36.956836    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:24:37.010800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.010708    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hgn\" (UniqueName: \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:37.011105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.011081    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s7t\" (UniqueName: \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.011316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.011297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.011509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.011496    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:37.012625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.012600    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:37.043156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.043112    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hgn\" (UniqueName: \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\") pod \"payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:37.049315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.049284    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:37.127846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.127802    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s7t\" (UniqueName: \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.135208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.128149    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.156755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.156442    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.161880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.161846    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s7t\" (UniqueName: \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\") pod \"payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.252625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.252593    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:37.460935 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.460901    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:24:37.464048 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:37.464020    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1625ec04_2af7_4911_8dfd_5cb34f4debc8.slice/crio-3694b80061f7b0aa7d6121a31cfb8ccef727c6ae3f228106f8e8f110b7469dda WatchSource:0}: Error finding container 3694b80061f7b0aa7d6121a31cfb8ccef727c6ae3f228106f8e8f110b7469dda: Status 404 returned error can't find the container with id 3694b80061f7b0aa7d6121a31cfb8ccef727c6ae3f228106f8e8f110b7469dda
Oct 30 03:24:37.591417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:37.591385    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:24:37.591567 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:37.591494    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b63c816_f9fe_4cb8_8d7f_1d1089f940d4.slice/crio-23ee05b68f0aa3f987fcdc5d4f0d093494f0b50a121a301888d031ff98feebf2 WatchSource:0}: Error finding container 23ee05b68f0aa3f987fcdc5d4f0d093494f0b50a121a301888d031ff98feebf2: Status 404 returned error can't find the container with id 23ee05b68f0aa3f987fcdc5d4f0d093494f0b50a121a301888d031ff98feebf2
Oct 30 03:24:38.355177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.355128    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" event={"ID":"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4","Type":"ContainerStarted","Data":"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"}
Oct 30 03:24:38.355706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.355200    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" event={"ID":"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4","Type":"ContainerStarted","Data":"23ee05b68f0aa3f987fcdc5d4f0d093494f0b50a121a301888d031ff98feebf2"}
Oct 30 03:24:38.355706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.355266    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:38.357320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.357286    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" event={"ID":"1625ec04-2af7-4911-8dfd-5cb34f4debc8","Type":"ContainerStarted","Data":"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"}
Oct 30 03:24:38.357500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.357365    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" event={"ID":"1625ec04-2af7-4911-8dfd-5cb34f4debc8","Type":"ContainerStarted","Data":"3694b80061f7b0aa7d6121a31cfb8ccef727c6ae3f228106f8e8f110b7469dda"}
Oct 30 03:24:38.357500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.357396    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:38.400604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.400550    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" podStartSLOduration=2.400536089 podStartE2EDuration="2.400536089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:38.397336995 +0000 UTC m=+2617.471507788" watchObservedRunningTime="2026-10-30 03:24:38.400536089 +0000 UTC m=+2617.474706936"
Oct 30 03:24:38.444401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.444347    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" podStartSLOduration=2.444333884 podStartE2EDuration="2.444333884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:38.44332565 +0000 UTC m=+2617.517496443" watchObservedRunningTime="2026-10-30 03:24:38.444333884 +0000 UTC m=+2617.518504677"
Oct 30 03:24:38.529843 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:38.529815    4214 logging.go:55] [core] [Channel #11557 SubChannel #11558]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.60:50052", ServerName: "10.133.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.60:50052: connect: connection refused"
Oct 30 03:24:38.829178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:38.829145    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 03:24:39.016590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.016503    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\") pod \"324658d1-42c7-4fba-90a7-7ff7b23de35c\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") "
Oct 30 03:24:39.016757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.016672    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\" (UniqueName: \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\") pod \"324658d1-42c7-4fba-90a7-7ff7b23de35c\" (UID: \"324658d1-42c7-4fba-90a7-7ff7b23de35c\") "
Oct 30 03:24:39.016933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.016904    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume" pod "324658d1-42c7-4fba-90a7-7ff7b23de35c" (UID: "324658d1-42c7-4fba-90a7-7ff7b23de35c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:39.017336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.017316    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324658d1-42c7-4fba-90a7-7ff7b23de35c-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:39.018892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.018863    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4" pod "324658d1-42c7-4fba-90a7-7ff7b23de35c" (UID: "324658d1-42c7-4fba-90a7-7ff7b23de35c"). InnerVolumeSpecName "kube-api-access-98mb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:39.119041 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.119001    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98mb4\" (UniqueName: \"kubernetes.io/projected/324658d1-42c7-4fba-90a7-7ff7b23de35c-kube-api-access-98mb4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:39.168045 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:39.168001    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:24:39.367758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.367719    4214 generic.go:356] "Generic (PLEG): container finished" podID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerID="97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80" exitCode=0
Oct 30 03:24:39.368311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.367796    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"
Oct 30 03:24:39.368311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.367816    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" event={"ID":"324658d1-42c7-4fba-90a7-7ff7b23de35c","Type":"ContainerDied","Data":"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"}
Oct 30 03:24:39.368311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.367890    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" event={"ID":"324658d1-42c7-4fba-90a7-7ff7b23de35c","Type":"ContainerDied","Data":"917956f19d4477642de21d9abbe43f1cae791087b06b9481a6de49fa13a47413"}
Oct 30 03:24:39.368311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.367918    4214 scope.go:122] "RemoveContainer" containerID="97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"
Oct 30 03:24:39.389771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.380660    4214 scope.go:122] "RemoveContainer" containerID="97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"
Oct 30 03:24:39.389771 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:39.381033    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80\": container with ID starting with 97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80 not found: ID does not exist" containerID="97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"
Oct 30 03:24:39.389771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.381065    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80"} err="failed to get container status \"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80\": rpc error: code = NotFound desc = could not find container \"97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80\": container with ID starting with 97e8c85eec0ace69112b8bafb0bd99793b4adcb7266a02662ca49bcdc2808e80 not found: ID does not exist"
Oct 30 03:24:39.426075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.426036    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 03:24:39.430157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.430130    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n"]
Oct 30 03:24:39.530545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:39.530458    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-810119-7985cbbff5-t7v5n" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.60:50052\" within 1s: context deadline exceeded"
Oct 30 03:24:39.530545 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:39.530512    4214 logging.go:55] [core] [Channel #11557 SubChannel #11558]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.60:50052", ServerName: "10.133.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.60:50052: operation was canceled"
Oct 30 03:24:41.076150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:41.076112    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" path="/var/lib/kubelet/pods/324658d1-42c7-4fba-90a7-7ff7b23de35c/volumes"
Oct 30 03:24:42.767598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.767562    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:24:42.768089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.767805    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy" containerID="cri-o://f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d" gracePeriod=30
Oct 30 03:24:42.777102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.777072    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:24:42.777356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.777326    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" containerID="cri-o://bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360" gracePeriod=30
Oct 30 03:24:42.783359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.783332    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:24:42.784367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.784341    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" containerID="cri-o://709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a" gracePeriod=30
Oct 30 03:24:42.787375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.787352    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:24:42.787483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:42.787404    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:43.660715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:43.660672    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:24:43.660893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:43.660732    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:47.336605 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.336572    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:24:47.337683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.337652    4214 cpu_manager.go:414] "containerMap: removing container" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing"
Oct 30 03:24:47.337683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.337680    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing"
Oct 30 03:24:47.337921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.337864    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="324658d1-42c7-4fba-90a7-7ff7b23de35c" containerName="payload-pre-processing"
Oct 30 03:24:47.343096 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.343074    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.362193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.362162    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e9528f-openshift-default-dockercfg-gtn85\""
Oct 30 03:24:47.367496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.367468    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381083    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381208    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381396    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381438    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381576    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381608    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381671    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwsj\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381728    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.382288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.381815    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489149    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489219    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489328    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489358    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489407    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwsj\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489487    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489568    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489607    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489677    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.489852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.489751    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.490183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.490160    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.490419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.490396    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.490516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.490495    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.490735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.490718    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.492150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.492126    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.492874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.492850    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.516200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.516171    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.516431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.516408    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwsj\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\") pod \"e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") " pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:47.657917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:47.657606    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:48.012893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.012864    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:24:48.014769 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:48.014732    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd1b954_857a_4b97_8e07_2fdbf90d9cd0.slice/crio-a1b031b0fa61329f8dafa87ef0fa2fdfdeac97fce401f6ef11a19eb45847b74a WatchSource:0}: Error finding container a1b031b0fa61329f8dafa87ef0fa2fdfdeac97fce401f6ef11a19eb45847b74a: Status 404 returned error can't find the container with id a1b031b0fa61329f8dafa87ef0fa2fdfdeac97fce401f6ef11a19eb45847b74a
Oct 30 03:24:48.020172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.020129    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:48.020294 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.020249    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:48.020375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.020301    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:24:48.063531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.063505    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:24:48.099396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.099362    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.099683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.099644    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.099806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.099706    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.099956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.099937    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.100051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100017    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.100298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100279    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.100437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100363    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.100538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100492    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.100943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100907    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.101095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.100986    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.101700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.101678    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.102152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.101902    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.102365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.102193    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\") pod \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\" (UID: \"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2\") "
Oct 30 03:24:48.102525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.102493    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.104220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.103558    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.104220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.103928    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.104220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.103950    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.104220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.103964    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.104220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.103978    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.120034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.120005    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.123535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.123508    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.123974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.123950    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.129360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.129323    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq" pod "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" (UID: "7cf2ede2-ce90-4134-b94d-80aa6a60a5a2"). InnerVolumeSpecName "kube-api-access-jszrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.152322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.152203    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:24:48.158708 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.158619    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:24:48.209764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.209527    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.209764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.209576    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.209764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.209612    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.209764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.209649    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.210416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.209690    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.210416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210207    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.210416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210259    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.210416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210308    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.210416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210388    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210846    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210918    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210966    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210991    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211112    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211153    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211219    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\") pod \"caa4eeb6-9813-4196-b508-69d3943c4f73\" (UID: \"caa4eeb6-9813-4196-b508-69d3943c4f73\") "
Oct 30 03:24:48.211780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211651    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.211780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211716    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\") pod \"7f75a085-ce38-40e8-be56-a8ddaebf1336\" (UID: \"7f75a085-ce38-40e8-be56-a8ddaebf1336\") "
Oct 30 03:24:48.213189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.213158    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.213403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.213351    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.213567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.213372    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.213567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.213523    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszrq\" (UniqueName: \"kubernetes.io/projected/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2-kube-api-access-jszrq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.210646    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.211204    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.215944    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.216210    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.216325    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.216725    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.217213    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.217469    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.217887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.217638    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.221201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.218860    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.221201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.220902    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.222036 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.221824    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.222290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.222212    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.222290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.222270    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.222557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.222243    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.223977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.223939    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj" pod "7f75a085-ce38-40e8-be56-a8ddaebf1336" (UID: "7f75a085-ce38-40e8-be56-a8ddaebf1336"). InnerVolumeSpecName "kube-api-access-lg5rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.226823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.226778    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "kube-api-access-pkq89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.226936 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.226917    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo" pod "caa4eeb6-9813-4196-b508-69d3943c4f73" (UID: "caa4eeb6-9813-4196-b508-69d3943c4f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.315477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315424    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315475    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315493    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa4eeb6-9813-4196-b508-69d3943c4f73-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315507    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315536    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315551    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315564    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315577    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315605    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315619    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315659    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315689    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5rj\" (UniqueName: \"kubernetes.io/projected/7f75a085-ce38-40e8-be56-a8ddaebf1336-kube-api-access-lg5rj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315707    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f75a085-ce38-40e8-be56-a8ddaebf1336-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315719    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.315738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315732    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq89\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-kube-api-access-pkq89\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.316418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315746    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.316418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315774    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f75a085-ce38-40e8-be56-a8ddaebf1336-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.316418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.315786    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa4eeb6-9813-4196-b508-69d3943c4f73-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.502024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.501935    4214 generic.go:356] "Generic (PLEG): container finished" podID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerID="f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d" exitCode=0
Oct 30 03:24:48.502024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.502000    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"
Oct 30 03:24:48.502563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.502018    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" event={"ID":"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2","Type":"ContainerDied","Data":"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"}
Oct 30 03:24:48.502563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.502077    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks" event={"ID":"7cf2ede2-ce90-4134-b94d-80aa6a60a5a2","Type":"ContainerDied","Data":"6c995fec3e0bd59b3e8addddf8e8a585204e44919ad114c8b8ce5dc4b105f05a"}
Oct 30 03:24:48.502563 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.502107    4214 scope.go:122] "RemoveContainer" containerID="f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"
Oct 30 03:24:48.504062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.504036    4214 generic.go:356] "Generic (PLEG): container finished" podID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerID="709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a" exitCode=0
Oct 30 03:24:48.504185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.504138    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"
Oct 30 03:24:48.504347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.504282    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" event={"ID":"7f75a085-ce38-40e8-be56-a8ddaebf1336","Type":"ContainerDied","Data":"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"}
Oct 30 03:24:48.504347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.504332    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w" event={"ID":"7f75a085-ce38-40e8-be56-a8ddaebf1336","Type":"ContainerDied","Data":"531b548efe405eb3b5eb7387e1dc577778a68c642f59e2fc2dba5386d50cabb8"}
Oct 30 03:24:48.510099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.509867    4214 generic.go:356] "Generic (PLEG): container finished" podID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerID="bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360" exitCode=0
Oct 30 03:24:48.510099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.509923    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"
Oct 30 03:24:48.510099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.510005    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" event={"ID":"caa4eeb6-9813-4196-b508-69d3943c4f73","Type":"ContainerDied","Data":"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"}
Oct 30 03:24:48.510099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.510049    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8" event={"ID":"caa4eeb6-9813-4196-b508-69d3943c4f73","Type":"ContainerDied","Data":"b925439076de311569ef107c3c2189d5cbcf3dd669bb9dead4e4d6a06a108034"}
Oct 30 03:24:48.519324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.519294    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerStarted","Data":"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"}
Oct 30 03:24:48.519450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.519343    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerStarted","Data":"a1b031b0fa61329f8dafa87ef0fa2fdfdeac97fce401f6ef11a19eb45847b74a"}
Oct 30 03:24:48.543296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.543269    4214 scope.go:122] "RemoveContainer" containerID="f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"
Oct 30 03:24:48.543631 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:48.543611    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d\": container with ID starting with f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d not found: ID does not exist" containerID="f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"
Oct 30 03:24:48.543716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.543639    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d"} err="failed to get container status \"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d\": rpc error: code = NotFound desc = could not find container \"f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d\": container with ID starting with f352322e4d491086f8c1c43aaf2b6e469d0d384f4fffc524fc2ed9212ee7ab0d not found: ID does not exist"
Oct 30 03:24:48.543716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.543656    4214 scope.go:122] "RemoveContainer" containerID="709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"
Oct 30 03:24:48.556859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.556837    4214 scope.go:122] "RemoveContainer" containerID="709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"
Oct 30 03:24:48.557194 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:48.557171    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a\": container with ID starting with 709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a not found: ID does not exist" containerID="709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"
Oct 30 03:24:48.557295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.557207    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a"} err="failed to get container status \"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a\": rpc error: code = NotFound desc = could not find container \"709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a\": container with ID starting with 709d0674157e296400f82c36762d21b617b89570e67f299c119f2d43560dbb3a not found: ID does not exist"
Oct 30 03:24:48.557295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.557244    4214 scope.go:122] "RemoveContainer" containerID="bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"
Oct 30 03:24:48.567655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.567636    4214 scope.go:122] "RemoveContainer" containerID="bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"
Oct 30 03:24:48.568044 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:48.568000    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360\": container with ID starting with bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360 not found: ID does not exist" containerID="bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"
Oct 30 03:24:48.568044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.568040    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360"} err="failed to get container status \"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360\": rpc error: code = NotFound desc = could not find container \"bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360\": container with ID starting with bdee3b03abfa0bd77414c3570c3bbe7fb2f5799cdf6af4a2335ea0041d962360 not found: ID does not exist"
Oct 30 03:24:48.595847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.595814    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:24:48.599753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.599726    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-rjrks"]
Oct 30 03:24:48.635197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.635162    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:24:48.639101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.639076    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-gcf5w"]
Oct 30 03:24:48.658342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.658308    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:48.680172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.680113    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podStartSLOduration=1.6800983710000001 podStartE2EDuration="1.680098371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:48.677827774 +0000 UTC m=+2627.751998568" watchObservedRunningTime="2026-10-30 03:24:48.680098371 +0000 UTC m=+2627.754269164"
Oct 30 03:24:48.726616 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.726577    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:24:48.733686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.733651    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-h82z8"]
Oct 30 03:24:48.930686 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.930648    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:24:48.931556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931529    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy"
Oct 30 03:24:48.931556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931555    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy"
Oct 30 03:24:48.931754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931583    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy"
Oct 30 03:24:48.931754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931593    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy"
Oct 30 03:24:48.931754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931612    4214 cpu_manager.go:414] "containerMap: removing container" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy"
Oct 30 03:24:48.931754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931620    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy"
Oct 30 03:24:48.931972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931847    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" containerName="istio-proxy"
Oct 30 03:24:48.931972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931861    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" containerName="istio-proxy"
Oct 30 03:24:48.931972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.931869    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" containerName="istio-proxy"
Oct 30 03:24:48.937685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.937665    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:48.955371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.955342    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-874ab5\"/\"openshift-service-ca.crt\""
Oct 30 03:24:48.955371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.955368    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-874ab5\"/\"kube-root-ca.crt\""
Oct 30 03:24:48.955660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.955639    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-874ab5\"/\"auth-test-model-874ab5-kserve-self-signed-certs\""
Oct 30 03:24:48.955722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.955682    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-874ab5\"/\"default-dockercfg-t5jrp\""
Oct 30 03:24:48.959394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:48.959372    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:24:49.034349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034310    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.034527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034490    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxxk\" (UniqueName: \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.034621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034588    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.034669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034636    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.034734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034718    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.034859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.034839    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.069090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.069056    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf2ede2-ce90-4134-b94d-80aa6a60a5a2" path="/var/lib/kubelet/pods/7cf2ede2-ce90-4134-b94d-80aa6a60a5a2/volumes"
Oct 30 03:24:49.069703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.069683    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f75a085-ce38-40e8-be56-a8ddaebf1336" path="/var/lib/kubelet/pods/7f75a085-ce38-40e8-be56-a8ddaebf1336/volumes"
Oct 30 03:24:49.070141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.070127    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa4eeb6-9813-4196-b508-69d3943c4f73" path="/var/lib/kubelet/pods/caa4eeb6-9813-4196-b508-69d3943c4f73/volumes"
Oct 30 03:24:49.136424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136391    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxxk\" (UniqueName: \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136466    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136507    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136622 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136542    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136634    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136705    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.136981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.136954    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.137223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.137204    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.137365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.137344    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.139199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.139164    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.139582 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.139556    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.166269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.166196    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxxk\" (UniqueName: \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\") pod \"auth-test-model-874ab5-kserve-c984bdf9c-qm7v8\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") " pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.253660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.253571    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:49.372412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.372389    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:24:49.374749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.374722    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:24:49.665604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.665567    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:49.671146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:49.671119    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:24:49.671833 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:49.671809    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod710722ea_d6f2_4c7e_844b_1ec0ef712f81.slice/crio-6ce5e4c6823b9be46a4a57c9e5e16406fb9f2d160a753aebb703b82f4043bc7a WatchSource:0}: Error finding container 6ce5e4c6823b9be46a4a57c9e5e16406fb9f2d160a753aebb703b82f4043bc7a: Status 404 returned error can't find the container with id 6ce5e4c6823b9be46a4a57c9e5e16406fb9f2d160a753aebb703b82f4043bc7a
Oct 30 03:24:50.550293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:50.550195    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" event={"ID":"710722ea-d6f2-4c7e-844b-1ec0ef712f81","Type":"ContainerStarted","Data":"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"}
Oct 30 03:24:50.550293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:50.550287    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:50.550594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:50.550319    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" event={"ID":"710722ea-d6f2-4c7e-844b-1ec0ef712f81","Type":"ContainerStarted","Data":"6ce5e4c6823b9be46a4a57c9e5e16406fb9f2d160a753aebb703b82f4043bc7a"}
Oct 30 03:24:50.551406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:50.551382    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:24:50.596126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:50.596061    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" podStartSLOduration=2.596042404 podStartE2EDuration="2.596042404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:50.592802334 +0000 UTC m=+2629.666973128" watchObservedRunningTime="2026-10-30 03:24:50.596042404 +0000 UTC m=+2629.670213197"
Oct 30 03:24:51.254805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:51.254763    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:53.248423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.248388    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:24:53.253329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.253305    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.271773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.271745    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:24:53.279617 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.279587    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9528f-dockercfg-cfkx2\""
Oct 30 03:24:53.279849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.279680    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9528f\""
Oct 30 03:24:53.292525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.292492    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:24:53.298875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.298387    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.328280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.324798    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.328280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.324927    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4k9\" (UniqueName: \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.335757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.334955    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:24:53.451264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.437174    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.451264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.437246    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpn2x\" (UniqueName: \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.451264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.437359    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4k9\" (UniqueName: \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.451264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.437447    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.451264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.438435    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.531304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.529842    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4k9\" (UniqueName: \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.538673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.538630    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpn2x\" (UniqueName: \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.538851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.538833    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.539931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.539901    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.563426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.563395    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpn2x\" (UniqueName: \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\") pod \"payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.570159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.570126    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:53.651149 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.636257    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:53.677486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.677454    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 03:24:53.891348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.891318    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:24:53.892096 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:53.892067    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53547826_3272_479b_8e84_e62933af4d27.slice/crio-9bd66e6c633c991343e780705a48e4136ce40e295a99015b139aeeef9558041a WatchSource:0}: Error finding container 9bd66e6c633c991343e780705a48e4136ce40e295a99015b139aeeef9558041a: Status 404 returned error can't find the container with id 9bd66e6c633c991343e780705a48e4136ce40e295a99015b139aeeef9558041a
Oct 30 03:24:53.989982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:53.989950    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:24:53.991816 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:24:53.991791    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode829d47a_f503_4fd4_b798_cd88d7aa67d7.slice/crio-1410b6d4d6d46746d7e497f0801512cda23d07a4aacc958ce10671df5d3612a5 WatchSource:0}: Error finding container 1410b6d4d6d46746d7e497f0801512cda23d07a4aacc958ce10671df5d3612a5: Status 404 returned error can't find the container with id 1410b6d4d6d46746d7e497f0801512cda23d07a4aacc958ce10671df5d3612a5
Oct 30 03:24:54.590434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.590398    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" event={"ID":"e829d47a-f503-4fd4-b798-cd88d7aa67d7","Type":"ContainerStarted","Data":"af09f1f961ed4d2f3e9f5e13c401b387be2b571fa96e5ee72cf4ff873aed5d0d"}
Oct 30 03:24:54.590957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.590457    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" event={"ID":"e829d47a-f503-4fd4-b798-cd88d7aa67d7","Type":"ContainerStarted","Data":"1410b6d4d6d46746d7e497f0801512cda23d07a4aacc958ce10671df5d3612a5"}
Oct 30 03:24:54.590957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.590490    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:24:54.592425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.592390    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" event={"ID":"53547826-3272-479b-8e84-e62933af4d27","Type":"ContainerStarted","Data":"f39a86727245215acd1586a5a25ef367257cbf7f767b66f3388467ab1fd31220"}
Oct 30 03:24:54.592560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.592431    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" event={"ID":"53547826-3272-479b-8e84-e62933af4d27","Type":"ContainerStarted","Data":"9bd66e6c633c991343e780705a48e4136ce40e295a99015b139aeeef9558041a"}
Oct 30 03:24:54.592560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.592452    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:24:54.636522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.636473    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" podStartSLOduration=1.6364594239999999 podStartE2EDuration="1.636459424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:54.635369442 +0000 UTC m=+2633.709540247" watchObservedRunningTime="2026-10-30 03:24:54.636459424 +0000 UTC m=+2633.710630207"
Oct 30 03:24:54.684424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:54.684360    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" podStartSLOduration=1.6843415529999999 podStartE2EDuration="1.684341553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:54.678971803 +0000 UTC m=+2633.753142597" watchObservedRunningTime="2026-10-30 03:24:54.684341553 +0000 UTC m=+2633.758512348"
Oct 30 03:24:58.678785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:58.678737    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" podUID="3d41e703-823a-4555-8aef-87719d078e7d" containerName="payload-processing" containerID="cri-o://3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250" gracePeriod=25
Oct 30 03:24:58.980985 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:58.980955    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 03:24:59.081620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.081589    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\") pod \"3d41e703-823a-4555-8aef-87719d078e7d\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") "
Oct 30 03:24:59.081794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.081675    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\" (UniqueName: \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\") pod \"3d41e703-823a-4555-8aef-87719d078e7d\" (UID: \"3d41e703-823a-4555-8aef-87719d078e7d\") "
Oct 30 03:24:59.082015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.081992    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume" pod "3d41e703-823a-4555-8aef-87719d078e7d" (UID: "3d41e703-823a-4555-8aef-87719d078e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:59.082393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.082364    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d41e703-823a-4555-8aef-87719d078e7d-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.084392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.084368    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx" pod "3d41e703-823a-4555-8aef-87719d078e7d" (UID: "3d41e703-823a-4555-8aef-87719d078e7d"). InnerVolumeSpecName "kube-api-access-g6bcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:59.184215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.184128    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6bcx\" (UniqueName: \"kubernetes.io/projected/3d41e703-823a-4555-8aef-87719d078e7d-kube-api-access-g6bcx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.254825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.254786    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:59.268970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.268941    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:59.685903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.685870    4214 generic.go:356] "Generic (PLEG): container finished" podID="3d41e703-823a-4555-8aef-87719d078e7d" containerID="3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250" exitCode=0
Oct 30 03:24:59.686395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.685915    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"
Oct 30 03:24:59.686395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.685951    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" event={"ID":"3d41e703-823a-4555-8aef-87719d078e7d","Type":"ContainerDied","Data":"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"}
Oct 30 03:24:59.686395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.686013    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx" event={"ID":"3d41e703-823a-4555-8aef-87719d078e7d","Type":"ContainerDied","Data":"3eb51ee48eb7c6856395176542aaa721943e428a180a810f915e8cd11b48ec29"}
Oct 30 03:24:59.686395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.686040    4214 scope.go:122] "RemoveContainer" containerID="3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"
Oct 30 03:24:59.697595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.697577    4214 scope.go:122] "RemoveContainer" containerID="3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"
Oct 30 03:24:59.697899 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:24:59.697878    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250\": container with ID starting with 3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250 not found: ID does not exist" containerID="3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"
Oct 30 03:24:59.697966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.697906    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250"} err="failed to get container status \"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250\": rpc error: code = NotFound desc = could not find container \"3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250\": container with ID starting with 3436cc93ce96c3332773236e0f51412ca47abd7b9170a057deff7ec120b68250 not found: ID does not exist"
Oct 30 03:24:59.700714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.700695    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:24:59.791978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.791951    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 03:24:59.796679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:24:59.796652    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7d1788-784cfd57f6-4v4rx"]
Oct 30 03:25:01.071269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:01.071220    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d41e703-823a-4555-8aef-87719d078e7d" path="/var/lib/kubelet/pods/3d41e703-823a-4555-8aef-87719d078e7d/volumes"
Oct 30 03:25:05.158971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.158930    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:25:05.160141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.160119    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3d41e703-823a-4555-8aef-87719d078e7d" containerName="payload-processing"
Oct 30 03:25:05.160188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.160145    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d41e703-823a-4555-8aef-87719d078e7d" containerName="payload-processing"
Oct 30 03:25:05.160385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.160370    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d41e703-823a-4555-8aef-87719d078e7d" containerName="payload-processing"
Oct 30 03:25:05.165965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.165944    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.183303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.182066    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:25:05.188543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.188513    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948965-dockercfg-9gvc4\""
Oct 30 03:25:05.188688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.188652    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948965\""
Oct 30 03:25:05.282039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.281991    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9vj\" (UniqueName: \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.282295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.282269    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.383898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.383865    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9vj\" (UniqueName: \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.384274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.384032    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.384976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.384947    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.404130 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.404104    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9vj\" (UniqueName: \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\") pod \"payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.486207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.486116    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:05.608264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.606007    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:25:05.608264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.606698    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:25:05.825778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:05.825748    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:25:05.825901 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:25:05.825750    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b69b063_32f7_4699_97cf_85a5e4a01f98.slice/crio-c90291007d5825f093d595f75b398c32c377d24fb910aa55d76abe11d677b618 WatchSource:0}: Error finding container c90291007d5825f093d595f75b398c32c377d24fb910aa55d76abe11d677b618: Status 404 returned error can't find the container with id c90291007d5825f093d595f75b398c32c377d24fb910aa55d76abe11d677b618
Oct 30 03:25:06.757893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:06.757848    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" event={"ID":"8b69b063-32f7-4699-97cf-85a5e4a01f98","Type":"ContainerStarted","Data":"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"}
Oct 30 03:25:06.758465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:06.757922    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" event={"ID":"8b69b063-32f7-4699-97cf-85a5e4a01f98","Type":"ContainerStarted","Data":"c90291007d5825f093d595f75b398c32c377d24fb910aa55d76abe11d677b618"}
Oct 30 03:25:06.758465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:06.757967    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:06.806407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:06.806349    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" podStartSLOduration=1.806334005 podStartE2EDuration="1.806334005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:06.806058764 +0000 UTC m=+2645.880229557" watchObservedRunningTime="2026-10-30 03:25:06.806334005 +0000 UTC m=+2645.880504798"
Oct 30 03:25:09.167408 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:25:09.167358    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:25:10.515297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.515264    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:25:10.515747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.515469    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" podUID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" containerName="maas-api" containerID="cri-o://9ff889cb818635641a2889d1ff0b253a86bf08496c56af5b2f80edbae7c3e70a" gracePeriod=30
Oct 30 03:25:10.807095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.807057    4214 generic.go:356] "Generic (PLEG): container finished" podID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" containerID="9ff889cb818635641a2889d1ff0b253a86bf08496c56af5b2f80edbae7c3e70a" exitCode=0
Oct 30 03:25:10.807305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.807129    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" event={"ID":"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1","Type":"ContainerDied","Data":"9ff889cb818635641a2889d1ff0b253a86bf08496c56af5b2f80edbae7c3e70a"}
Oct 30 03:25:10.835979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.835952    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:25:10.976281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.976245    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\") pod \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") "
Oct 30 03:25:10.976510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.976431    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\") pod \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") "
Oct 30 03:25:10.976510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.976490    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\") pod \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") "
Oct 30 03:25:10.976636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.976594    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\" (UniqueName: \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\") pod \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\" (UID: \"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1\") "
Oct 30 03:25:10.976962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.976828    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca" pod "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" (UID: "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:10.977295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.977185    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:10.978576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.978553    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls" pod "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" (UID: "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:10.979066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.979024    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs" pod "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" (UID: "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:10.979165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:10.979145    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf" pod "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" (UID: "75d6bf98-c3dd-49f1-8c9f-97cee84c25f1"). InnerVolumeSpecName "kube-api-access-wrwpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:11.079355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.078412    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.079355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.078448    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.079355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.078465    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrwpf\" (UniqueName: \"kubernetes.io/projected/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1-kube-api-access-wrwpf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.312620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.312579    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:25:11.313810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.313786    4214 cpu_manager.go:414] "containerMap: removing container" podUID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" containerName="maas-api"
Oct 30 03:25:11.313879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.313815    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" containerName="maas-api"
Oct 30 03:25:11.314068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.314053    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" containerName="maas-api"
Oct 30 03:25:11.318838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.318815    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.338790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.338721    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-8b0787-serving-cert\""
Oct 30 03:25:11.341266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.341222    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:25:11.391510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.391475    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.391709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.391534    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.391709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.391657    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.391843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.391749    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bjb\" (UniqueName: \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.493818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.493763    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.494060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.493846    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.494060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.493979    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.494222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.494079    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bjb\" (UniqueName: \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.495126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.495095    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.497360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.497332    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.497493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.497454    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.521388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.521367    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bjb\" (UniqueName: \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\") pod \"maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.635710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.635623    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:11.818054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.818023    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"
Oct 30 03:25:11.818240 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.818050    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl" event={"ID":"75d6bf98-c3dd-49f1-8c9f-97cee84c25f1","Type":"ContainerDied","Data":"7d68ed5d396b33c520fb5e6000091e5b6696dd3495e9a89958b61e7dd1bb4f2b"}
Oct 30 03:25:11.818240 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.818115    4214 scope.go:122] "RemoveContainer" containerID="9ff889cb818635641a2889d1ff0b253a86bf08496c56af5b2f80edbae7c3e70a"
Oct 30 03:25:11.877871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.877839    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:25:11.882281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.882254    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-086aee-5d4f6b4db6-hxtcl"]
Oct 30 03:25:11.974433 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:25:11.974407    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f09ee_73ed_49a9_8ec6_6e0f96982882.slice/crio-0ef665a948faaa7e2c92b3903dec12a9e8f1dcfc2da8fa41c139192f3f2afc54 WatchSource:0}: Error finding container 0ef665a948faaa7e2c92b3903dec12a9e8f1dcfc2da8fa41c139192f3f2afc54: Status 404 returned error can't find the container with id 0ef665a948faaa7e2c92b3903dec12a9e8f1dcfc2da8fa41c139192f3f2afc54
Oct 30 03:25:11.974656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.974630    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:25:11.976196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:11.976175    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:25:12.828300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:12.828264    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" event={"ID":"3b2f09ee-73ed-49a9-8ec6-6e0f96982882","Type":"ContainerStarted","Data":"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"}
Oct 30 03:25:12.828782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:12.828326    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" event={"ID":"3b2f09ee-73ed-49a9-8ec6-6e0f96982882","Type":"ContainerStarted","Data":"0ef665a948faaa7e2c92b3903dec12a9e8f1dcfc2da8fa41c139192f3f2afc54"}
Oct 30 03:25:12.828782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:12.828374    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:12.891442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:12.891387    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" podStartSLOduration=1.510078632 podStartE2EDuration="1.891372147s" totalImagesPullingTime="381.293515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:12.888149583 +0000 UTC m=+2651.962320376" watchObservedRunningTime="2026-10-30 03:25:12.891372147 +0000 UTC m=+2651.965542939"
Oct 30 03:25:13.069018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:13.068988    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d6bf98-c3dd-49f1-8c9f-97cee84c25f1" path="/var/lib/kubelet/pods/75d6bf98-c3dd-49f1-8c9f-97cee84c25f1/volumes"
Oct 30 03:25:17.772178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:17.772094    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:25:18.852933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:18.852900    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:25:20.479689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:20.479653    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:25:20.903599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:20.903553    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy" containerID="cri-o://45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923" gracePeriod=30
Oct 30 03:25:26.211436 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.211411    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:25:26.370746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.370611    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.377529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.376882    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380599    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380694    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380733    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380787    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380832    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.380975    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.381016    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.381207    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\") pod \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\" (UID: \"83887e84-0d2e-46e9-b4e7-8e3cb5636160\") "
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.389083    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.389279    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:26.389287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.389293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:26.390046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.389624    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "kube-api-access-82wb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:26.390256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.390187    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:26.391927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.391888    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:26.392127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.392109    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:25:26.393598 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.393562    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token" pod "83887e84-0d2e-46e9-b4e7-8e3cb5636160" (UID: "83887e84-0d2e-46e9-b4e7-8e3cb5636160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483172    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483205    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wb4\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-kube-api-access-82wb4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483220    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483257    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483272    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483282    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483294    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483308    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.483678 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.483319    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83887e84-0d2e-46e9-b4e7-8e3cb5636160-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:26.973150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.973113    4214 generic.go:356] "Generic (PLEG): container finished" podID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerID="45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923" exitCode=0
Oct 30 03:25:26.973410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.973178    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"
Oct 30 03:25:26.973410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.973210    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerDied","Data":"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"}
Oct 30 03:25:26.973410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.973294    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8" event={"ID":"83887e84-0d2e-46e9-b4e7-8e3cb5636160","Type":"ContainerDied","Data":"6956f25ea87e9385d2c596f21ff43f116d17f799219881132e1ddde07cf8b885"}
Oct 30 03:25:26.973410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:26.973325    4214 scope.go:122] "RemoveContainer" containerID="45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"
Oct 30 03:25:27.002095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.002038    4214 scope.go:122] "RemoveContainer" containerID="c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"
Oct 30 03:25:27.012643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.012620    4214 scope.go:122] "RemoveContainer" containerID="45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"
Oct 30 03:25:27.012932 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:25:27.012906    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923\": container with ID starting with 45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923 not found: ID does not exist" containerID="45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"
Oct 30 03:25:27.012987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.012942    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923"} err="failed to get container status \"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923\": rpc error: code = NotFound desc = could not find container \"45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923\": container with ID starting with 45f399eedc6d42eb914a4230b716b0e6855093bc2ad65064680c0a8c51e8d923 not found: ID does not exist"
Oct 30 03:25:27.012987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.012968    4214 scope.go:122] "RemoveContainer" containerID="c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"
Oct 30 03:25:27.013281 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:25:27.013244    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655\": container with ID starting with c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655 not found: ID does not exist" containerID="c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"
Oct 30 03:25:27.013390 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.013287    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655"} err="failed to get container status \"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655\": rpc error: code = NotFound desc = could not find container \"c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655\": container with ID starting with c9c80832029ce1cde57a33c9dcbadf80a3d65987177a90d20273049c31001655 not found: ID does not exist"
Oct 30 03:25:27.075822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.075783    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:25:27.088660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.088623    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-086aee-openshift-default-87d5ddd45-k76z8"]
Oct 30 03:25:27.177695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:27.177661    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w_8b69b063-32f7-4699-97cf-85a5e4a01f98/payload-processing/0.log"
Oct 30 03:25:29.069623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:29.069588    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" path="/var/lib/kubelet/pods/83887e84-0d2e-46e9-b4e7-8e3cb5636160/volumes"
Oct 30 03:25:39.166509 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:25:39.166470    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:25:42.122305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.122276    4214 generic.go:356] "Generic (PLEG): container finished" podID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerID="06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7" exitCode=0
Oct 30 03:25:42.122738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.122344    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerDied","Data":"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"}
Oct 30 03:25:42.122786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.122772    4214 scope.go:122] "RemoveContainer" containerID="06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"
Oct 30 03:25:42.124454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.124426    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:42.124531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.124497    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:42.124531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:42.124527    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:25:43.131468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:43.131435    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerStarted","Data":"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"}
Oct 30 03:25:43.950412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:43.950329    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:25:43.950716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:43.950693    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:25:43.950868 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:43.950837    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:25:44.955255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:44.953248    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:25:44.955255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:44.953308    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:25:45.960251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:45.956605    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:25:45.960251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:45.956663    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:25:46.954254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:46.953322    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:25:46.954254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:46.953396    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:25:47.297509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:47.297471    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:25:47.966143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:47.965648    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:25:47.966143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:47.965704    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:25:48.951989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:48.951954    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:25:48.953546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:48.953517    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:25:58.924218 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:58.924141    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:25:58.924655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:58.924376    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" podUID="e19faf67-97c9-49d4-a80b-a32dc8c42204" containerName="payload-pre-processing" containerID="cri-o://e864afa9e2f9e9cb9eb805e30e30e6f6c4bdd7f43d4ceea47480b5b059babeb5" gracePeriod=30
Oct 30 03:25:59.366139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.365859    4214 generic.go:356] "Generic (PLEG): container finished" podID="e19faf67-97c9-49d4-a80b-a32dc8c42204" containerID="e864afa9e2f9e9cb9eb805e30e30e6f6c4bdd7f43d4ceea47480b5b059babeb5" exitCode=0
Oct 30 03:25:59.366139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.366030    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" event={"ID":"e19faf67-97c9-49d4-a80b-a32dc8c42204","Type":"ContainerDied","Data":"e864afa9e2f9e9cb9eb805e30e30e6f6c4bdd7f43d4ceea47480b5b059babeb5"}
Oct 30 03:25:59.366139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.366083    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns" event={"ID":"e19faf67-97c9-49d4-a80b-a32dc8c42204","Type":"ContainerDied","Data":"12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c"}
Oct 30 03:25:59.366139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.366107    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12c229c690a4c6cf73e7d3ac116613e5d226056caee9088a2df2813412498c0c"
Oct 30 03:25:59.373741 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.373411    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:25:59.381254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.380318    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\") pod \"e19faf67-97c9-49d4-a80b-a32dc8c42204\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") "
Oct 30 03:25:59.381254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.380445    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\" (UniqueName: \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\") pod \"e19faf67-97c9-49d4-a80b-a32dc8c42204\" (UID: \"e19faf67-97c9-49d4-a80b-a32dc8c42204\") "
Oct 30 03:25:59.381938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.381900    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume" pod "e19faf67-97c9-49d4-a80b-a32dc8c42204" (UID: "e19faf67-97c9-49d4-a80b-a32dc8c42204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:59.389323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.389285    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt" pod "e19faf67-97c9-49d4-a80b-a32dc8c42204" (UID: "e19faf67-97c9-49d4-a80b-a32dc8c42204"). InnerVolumeSpecName "kube-api-access-xqtgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:59.482787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.482753    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19faf67-97c9-49d4-a80b-a32dc8c42204-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.482787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:25:59.482784    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqtgt\" (UniqueName: \"kubernetes.io/projected/e19faf67-97c9-49d4-a80b-a32dc8c42204-kube-api-access-xqtgt\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:00.388665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:00.385819    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"
Oct 30 03:26:00.444194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:00.444156    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:26:00.456829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:00.456794    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-767f74dd9f-d7nns"]
Oct 30 03:26:01.071397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:01.071353    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19faf67-97c9-49d4-a80b-a32dc8c42204" path="/var/lib/kubelet/pods/e19faf67-97c9-49d4-a80b-a32dc8c42204/volumes"
Oct 30 03:26:03.429124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.429082    4214 generic.go:356] "Generic (PLEG): container finished" podID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerID="50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f" exitCode=0
Oct 30 03:26:03.429689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.429134    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerDied","Data":"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"}
Oct 30 03:26:03.429689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.429664    4214 scope.go:122] "RemoveContainer" containerID="50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"
Oct 30 03:26:03.432053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.432014    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:03.432192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.432110    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:03.432192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:03.432157    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:04.444913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:04.444876    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerStarted","Data":"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"}
Oct 30 03:26:04.660657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:04.660572    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:26:04.664677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:04.664348    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:04.664677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:04.664407    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:05.555253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:05.554655    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:26:05.665252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:05.665060    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:05.665252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:05.665124    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:06.006845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:06.006750    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:26:06.007560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:06.007535    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy" containerID="cri-o://0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed" gracePeriod=30
Oct 30 03:26:06.660591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:06.660550    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:06.661210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:06.661180    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:07.500424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:07.500348    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:26:07.500807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:07.500783    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy" containerID="cri-o://66c1729bc589a7a60fc4d748d0b3340b7952cac8ba83aedee840e5b0b13886f8" gracePeriod=30
Oct 30 03:26:07.678250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:07.661673    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:07.678250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:07.661735    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:08.401027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:08.400987    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:26:08.467723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:08.467671    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:26:08.660697 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:08.660605    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:08.660996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:08.660961    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:09.317053 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:09.317011    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:26:09.671298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:09.670350    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Oct 30 03:26:09.671298 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:09.670412    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Oct 30 03:26:10.665076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:10.665042    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:26:10.666593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:10.666566    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:26:11.330140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.330114    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:26:11.441721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.441934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441739    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.441934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441766    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.441934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441809    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.441934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441842    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.441934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441871    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441947    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.441990    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442113    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442195    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:11.442260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442210    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\") pod \"df32909d-505e-4f7a-8afa-8ebfb3af15de\" (UID: \"df32909d-505e-4f7a-8afa-8ebfb3af15de\") "
Oct 30 03:26:11.442620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442520    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:11.442969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442728    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:11.442969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442953    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.443166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442975    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.443166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.442991    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df32909d-505e-4f7a-8afa-8ebfb3af15de-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.443166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.443009    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.443166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.443023    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.444403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.444375    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:11.444795 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.444764    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:11.445135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.445117    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:11.445215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.445144    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86" pod "df32909d-505e-4f7a-8afa-8ebfb3af15de" (UID: "df32909d-505e-4f7a-8afa-8ebfb3af15de"). InnerVolumeSpecName "kube-api-access-pfk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:11.544490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.544458    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.544490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.544490    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.544490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.544500    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df32909d-505e-4f7a-8afa-8ebfb3af15de-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.544822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.544510    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfk86\" (UniqueName: \"kubernetes.io/projected/df32909d-505e-4f7a-8afa-8ebfb3af15de-kube-api-access-pfk86\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:11.635082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.635032    4214 generic.go:356] "Generic (PLEG): container finished" podID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerID="0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed" exitCode=0
Oct 30 03:26:11.635281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.635093    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerDied","Data":"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"}
Oct 30 03:26:11.635281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.635156    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h" event={"ID":"df32909d-505e-4f7a-8afa-8ebfb3af15de","Type":"ContainerDied","Data":"1e6fe937af6682ef9b7682e8e91c34f568a71567e709f86f9bdb0e8ebed9a38c"}
Oct 30 03:26:11.635281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.635176    4214 scope.go:122] "RemoveContainer" containerID="0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"
Oct 30 03:26:11.635281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.635103    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"
Oct 30 03:26:11.647912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.647893    4214 scope.go:122] "RemoveContainer" containerID="84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"
Oct 30 03:26:11.659174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.659152    4214 scope.go:122] "RemoveContainer" containerID="0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"
Oct 30 03:26:11.659532 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:11.659505    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed\": container with ID starting with 0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed not found: ID does not exist" containerID="0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"
Oct 30 03:26:11.659637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.659541    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed"} err="failed to get container status \"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed\": rpc error: code = NotFound desc = could not find container \"0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed\": container with ID starting with 0cad098d20e2e6b7f48321364d7a7f4e5efc6c00fd6c13131438d8718edba6ed not found: ID does not exist"
Oct 30 03:26:11.659637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.659562    4214 scope.go:122] "RemoveContainer" containerID="84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"
Oct 30 03:26:11.659830 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:11.659810    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f\": container with ID starting with 84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f not found: ID does not exist" containerID="84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"
Oct 30 03:26:11.659872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.659838    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f"} err="failed to get container status \"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f\": rpc error: code = NotFound desc = could not find container \"84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f\": container with ID starting with 84623a6d0deb1af4324c4cde1744d1504cf8eb554a1f3afbc7e57b4cc5b0ff3f not found: ID does not exist"
Oct 30 03:26:11.692943 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.692907    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:26:11.696834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:11.696808    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-146bb7-openshift-default-79578d4855-d8f8h"]
Oct 30 03:26:12.645680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.645648    4214 generic.go:356] "Generic (PLEG): container finished" podID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerID="66c1729bc589a7a60fc4d748d0b3340b7952cac8ba83aedee840e5b0b13886f8" exitCode=0
Oct 30 03:26:12.645914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.645681    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerDied","Data":"66c1729bc589a7a60fc4d748d0b3340b7952cac8ba83aedee840e5b0b13886f8"}
Oct 30 03:26:12.645914 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.645759    4214 scope.go:122] "RemoveContainer" containerID="5a3ca86dfb6a0bcfe1cbe3ab2d85ef7fbe13b76938bd42d357d7bc0e125738ff"
Oct 30 03:26:12.831661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.831636    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:26:12.975405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975370    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975502    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975547    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975627    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975654    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975725    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975826    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.975894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975759    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:12.975894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975868    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.976072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.975918    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\") pod \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\" (UID: \"3ceb60bb-aae7-469c-a7f5-015f5a610c1f\") "
Oct 30 03:26:12.976072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976009    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:12.976196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976124    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:12.976293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976170    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:12.976293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976246    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:12.976635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976610    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.976757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976637    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.976757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976653    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.976757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976665    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.976757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.976681    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:12.978529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.978493    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:12.978680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.978639    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:12.978680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.978665    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:12.980019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:12.979996    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9" pod "3ceb60bb-aae7-469c-a7f5-015f5a610c1f" (UID: "3ceb60bb-aae7-469c-a7f5-015f5a610c1f"). InnerVolumeSpecName "kube-api-access-s9wj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:13.069543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.069466    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" path="/var/lib/kubelet/pods/df32909d-505e-4f7a-8afa-8ebfb3af15de/volumes"
Oct 30 03:26:13.077378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.077350    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wj9\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-kube-api-access-s9wj9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.077378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.077373    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.077378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.077383    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.077567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.077392    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb60bb-aae7-469c-a7f5-015f5a610c1f-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.402345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.402263    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" podUID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" containerName="payload-processing" containerID="cri-o://af09f1f961ed4d2f3e9f5e13c401b387be2b571fa96e5ee72cf4ff873aed5d0d" gracePeriod=25
Oct 30 03:26:13.468702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.468664    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" podUID="53547826-3272-479b-8e84-e62933af4d27" containerName="payload-pre-processing" containerID="cri-o://f39a86727245215acd1586a5a25ef367257cbf7f767b66f3388467ab1fd31220" gracePeriod=25
Oct 30 03:26:13.665258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.664989    4214 generic.go:356] "Generic (PLEG): container finished" podID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" containerID="af09f1f961ed4d2f3e9f5e13c401b387be2b571fa96e5ee72cf4ff873aed5d0d" exitCode=0
Oct 30 03:26:13.665258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.665129    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" event={"ID":"e829d47a-f503-4fd4-b798-cd88d7aa67d7","Type":"ContainerDied","Data":"af09f1f961ed4d2f3e9f5e13c401b387be2b571fa96e5ee72cf4ff873aed5d0d"}
Oct 30 03:26:13.670309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.667283    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs" event={"ID":"3ceb60bb-aae7-469c-a7f5-015f5a610c1f","Type":"ContainerDied","Data":"84f8396f618fb236371c2672af1a39e69b4d63912b067f8873824961df715ed0"}
Oct 30 03:26:13.670309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.667339    4214 scope.go:122] "RemoveContainer" containerID="66c1729bc589a7a60fc4d748d0b3340b7952cac8ba83aedee840e5b0b13886f8"
Oct 30 03:26:13.670309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.667500    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"
Oct 30 03:26:13.681161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.680006    4214 generic.go:356] "Generic (PLEG): container finished" podID="53547826-3272-479b-8e84-e62933af4d27" containerID="f39a86727245215acd1586a5a25ef367257cbf7f767b66f3388467ab1fd31220" exitCode=0
Oct 30 03:26:13.681161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.680103    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" event={"ID":"53547826-3272-479b-8e84-e62933af4d27","Type":"ContainerDied","Data":"f39a86727245215acd1586a5a25ef367257cbf7f767b66f3388467ab1fd31220"}
Oct 30 03:26:13.751389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.751327    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:26:13.759397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.759311    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a0f7e-openshift-default-5d9c577757-td9qs"]
Oct 30 03:26:13.769445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.769423    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:26:13.819564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.819542    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:26:13.886957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.886923    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\" (UniqueName: \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\") pod \"53547826-3272-479b-8e84-e62933af4d27\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") "
Oct 30 03:26:13.887459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.887144    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\") pod \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") "
Oct 30 03:26:13.887459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.887183    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\" (UniqueName: \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\") pod \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\" (UID: \"e829d47a-f503-4fd4-b798-cd88d7aa67d7\") "
Oct 30 03:26:13.887459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.887340    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\") pod \"53547826-3272-479b-8e84-e62933af4d27\" (UID: \"53547826-3272-479b-8e84-e62933af4d27\") "
Oct 30 03:26:13.887689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.887659    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume" pod "e829d47a-f503-4fd4-b798-cd88d7aa67d7" (UID: "e829d47a-f503-4fd4-b798-cd88d7aa67d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:13.887754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.887696    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume" pod "53547826-3272-479b-8e84-e62933af4d27" (UID: "53547826-3272-479b-8e84-e62933af4d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:13.888022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.888001    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e829d47a-f503-4fd4-b798-cd88d7aa67d7-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.888067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.888031    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53547826-3272-479b-8e84-e62933af4d27-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.889919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.889895    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x" pod "e829d47a-f503-4fd4-b798-cd88d7aa67d7" (UID: "e829d47a-f503-4fd4-b798-cd88d7aa67d7"). InnerVolumeSpecName "kube-api-access-bpn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:13.890059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.890021    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9" pod "53547826-3272-479b-8e84-e62933af4d27" (UID: "53547826-3272-479b-8e84-e62933af4d27"). InnerVolumeSpecName "kube-api-access-6n4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:13.989990 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.989916    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpn2x\" (UniqueName: \"kubernetes.io/projected/e829d47a-f503-4fd4-b798-cd88d7aa67d7-kube-api-access-bpn2x\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:13.989990 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:13.989941    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4k9\" (UniqueName: \"kubernetes.io/projected/53547826-3272-479b-8e84-e62933af4d27-kube-api-access-6n4k9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:14.693693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.693655    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"
Oct 30 03:26:14.693873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.693754    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8" event={"ID":"53547826-3272-479b-8e84-e62933af4d27","Type":"ContainerDied","Data":"9bd66e6c633c991343e780705a48e4136ce40e295a99015b139aeeef9558041a"}
Oct 30 03:26:14.693873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.693823    4214 scope.go:122] "RemoveContainer" containerID="f39a86727245215acd1586a5a25ef367257cbf7f767b66f3388467ab1fd31220"
Oct 30 03:26:14.696900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.696874    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn" event={"ID":"e829d47a-f503-4fd4-b798-cd88d7aa67d7","Type":"ContainerDied","Data":"1410b6d4d6d46746d7e497f0801512cda23d07a4aacc958ce10671df5d3612a5"}
Oct 30 03:26:14.697025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.696971    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"
Oct 30 03:26:14.706996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.706978    4214 scope.go:122] "RemoveContainer" containerID="af09f1f961ed4d2f3e9f5e13c401b387be2b571fa96e5ee72cf4ff873aed5d0d"
Oct 30 03:26:14.754504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.754471    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:26:14.766253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.766208    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-dfff754fd-s8kdn"]
Oct 30 03:26:14.790602 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.790572    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:26:14.794195 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:14.794173    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-6d94c9845f-dwvb8"]
Oct 30 03:26:15.069786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:15.069712    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" path="/var/lib/kubelet/pods/3ceb60bb-aae7-469c-a7f5-015f5a610c1f/volumes"
Oct 30 03:26:15.070147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:15.070135    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53547826-3272-479b-8e84-e62933af4d27" path="/var/lib/kubelet/pods/53547826-3272-479b-8e84-e62933af4d27/volumes"
Oct 30 03:26:15.070547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:15.070532    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" path="/var/lib/kubelet/pods/e829d47a-f503-4fd4-b798-cd88d7aa67d7/volumes"
Oct 30 03:26:18.440480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:18.440440    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:26:23.441416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.441367    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" podUID="8736cfd8-158b-407c-a473-7755bbb10ce2" containerName="payload-pre-processing" containerID="cri-o://8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed" gracePeriod=25
Oct 30 03:26:23.752223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.751888    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:26:23.782798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.782754    4214 generic.go:356] "Generic (PLEG): container finished" podID="8736cfd8-158b-407c-a473-7755bbb10ce2" containerID="8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed" exitCode=0
Oct 30 03:26:23.782971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.782800    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"
Oct 30 03:26:23.782971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.782823    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" event={"ID":"8736cfd8-158b-407c-a473-7755bbb10ce2","Type":"ContainerDied","Data":"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"}
Oct 30 03:26:23.782971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.782888    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98" event={"ID":"8736cfd8-158b-407c-a473-7755bbb10ce2","Type":"ContainerDied","Data":"8e2017ba72ae818add58eb8159c0f793545b307bfd930c8d8d363b1257cd6b14"}
Oct 30 03:26:23.782971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.782920    4214 scope.go:122] "RemoveContainer" containerID="8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"
Oct 30 03:26:23.793894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.793748    4214 scope.go:122] "RemoveContainer" containerID="8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"
Oct 30 03:26:23.794066 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:23.794045    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed\": container with ID starting with 8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed not found: ID does not exist" containerID="8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"
Oct 30 03:26:23.794113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.794079    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed"} err="failed to get container status \"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed\": rpc error: code = NotFound desc = could not find container \"8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed\": container with ID starting with 8b55c3faaabe6be7f3f5e8539d9d328cc937337200bca5d9d1e8e4f78cf3c0ed not found: ID does not exist"
Oct 30 03:26:23.933950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.933920    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\" (UniqueName: \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\") pod \"8736cfd8-158b-407c-a473-7755bbb10ce2\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") "
Oct 30 03:26:23.934143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.934129    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\") pod \"8736cfd8-158b-407c-a473-7755bbb10ce2\" (UID: \"8736cfd8-158b-407c-a473-7755bbb10ce2\") "
Oct 30 03:26:23.934550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.934519    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume" pod "8736cfd8-158b-407c-a473-7755bbb10ce2" (UID: "8736cfd8-158b-407c-a473-7755bbb10ce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:23.934774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.934757    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8736cfd8-158b-407c-a473-7755bbb10ce2-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:23.936316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:23.936288    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm" pod "8736cfd8-158b-407c-a473-7755bbb10ce2" (UID: "8736cfd8-158b-407c-a473-7755bbb10ce2"). InnerVolumeSpecName "kube-api-access-gsshm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:24.036531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:24.036444    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsshm\" (UniqueName: \"kubernetes.io/projected/8736cfd8-158b-407c-a473-7755bbb10ce2-kube-api-access-gsshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:24.137099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:24.137065    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:26:24.143084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:24.143052    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-723ffc-94746cdbc-v9d98"]
Oct 30 03:26:25.072651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:25.072615    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8736cfd8-158b-407c-a473-7755bbb10ce2" path="/var/lib/kubelet/pods/8736cfd8-158b-407c-a473-7755bbb10ce2/volumes"
Oct 30 03:26:34.526353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.526318    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:26:34.527409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.526533    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" podUID="014aa7e5-12d6-49e4-b052-adc27bc226a4" containerName="maas-api" containerID="cri-o://f87b2aa220743c7c8c1ca79d454b30ba453fa492c47a9606c3a59e8eed7f88f6" gracePeriod=30
Oct 30 03:26:34.873736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.873701    4214 generic.go:356] "Generic (PLEG): container finished" podID="014aa7e5-12d6-49e4-b052-adc27bc226a4" containerID="f87b2aa220743c7c8c1ca79d454b30ba453fa492c47a9606c3a59e8eed7f88f6" exitCode=0
Oct 30 03:26:34.873897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.873780    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" event={"ID":"014aa7e5-12d6-49e4-b052-adc27bc226a4","Type":"ContainerDied","Data":"f87b2aa220743c7c8c1ca79d454b30ba453fa492c47a9606c3a59e8eed7f88f6"}
Oct 30 03:26:34.873897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.873850    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m" event={"ID":"014aa7e5-12d6-49e4-b052-adc27bc226a4","Type":"ContainerDied","Data":"32951a7c326374f8c12a4e9892aacc0cfd13e73ed29d3804c443af517d0f7ee4"}
Oct 30 03:26:34.873897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.873874    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32951a7c326374f8c12a4e9892aacc0cfd13e73ed29d3804c443af517d0f7ee4"
Oct 30 03:26:34.883915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:34.883878    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:26:35.083251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.082939    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\") pod \"014aa7e5-12d6-49e4-b052-adc27bc226a4\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") "
Oct 30 03:26:35.083251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.083095    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\" (UniqueName: \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\") pod \"014aa7e5-12d6-49e4-b052-adc27bc226a4\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") "
Oct 30 03:26:35.083251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.083134    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\") pod \"014aa7e5-12d6-49e4-b052-adc27bc226a4\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") "
Oct 30 03:26:35.083251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.083172    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") pod \"014aa7e5-12d6-49e4-b052-adc27bc226a4\" (UID: \"014aa7e5-12d6-49e4-b052-adc27bc226a4\") "
Oct 30 03:26:35.083873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.083845    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca" pod "014aa7e5-12d6-49e4-b052-adc27bc226a4" (UID: "014aa7e5-12d6-49e4-b052-adc27bc226a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:35.086013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.085989    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28" pod "014aa7e5-12d6-49e4-b052-adc27bc226a4" (UID: "014aa7e5-12d6-49e4-b052-adc27bc226a4"). InnerVolumeSpecName "kube-api-access-k2p28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:35.086262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.086217    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls" pod "014aa7e5-12d6-49e4-b052-adc27bc226a4" (UID: "014aa7e5-12d6-49e4-b052-adc27bc226a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:35.087279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.087257    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs" pod "014aa7e5-12d6-49e4-b052-adc27bc226a4" (UID: "014aa7e5-12d6-49e4-b052-adc27bc226a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:35.185155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.185121    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:35.185155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.185157    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014aa7e5-12d6-49e4-b052-adc27bc226a4-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:35.185364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.185171    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2p28\" (UniqueName: \"kubernetes.io/projected/014aa7e5-12d6-49e4-b052-adc27bc226a4-kube-api-access-k2p28\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:35.185364 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.185185    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014aa7e5-12d6-49e4-b052-adc27bc226a4-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:35.882744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.882711    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"
Oct 30 03:26:35.938673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.938636    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:26:35.950775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:35.950748    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-723ffc-67776cff77-cx54m"]
Oct 30 03:26:37.070557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:37.070513    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014aa7e5-12d6-49e4-b052-adc27bc226a4" path="/var/lib/kubelet/pods/014aa7e5-12d6-49e4-b052-adc27bc226a4/volumes"
Oct 30 03:26:39.165464 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:39.165421    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:26:44.894853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:44.894758    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:26:44.895485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:44.895017    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" podUID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" containerName="maas-api" containerID="cri-o://a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560" gracePeriod=30
Oct 30 03:26:45.073715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.073680    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:26:45.073972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.073927    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" podUID="8b69b063-32f7-4699-97cf-85a5e4a01f98" containerName="payload-processing" containerID="cri-o://ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19" gracePeriod=30
Oct 30 03:26:45.227626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.227596    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:26:45.332250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.331756    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\") pod \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") "
Oct 30 03:26:45.332250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.332082    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\") pod \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") "
Oct 30 03:26:45.332250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.332153    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\") pod \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") "
Oct 30 03:26:45.332250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.332185    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\" (UniqueName: \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\") pod \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\" (UID: \"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe\") "
Oct 30 03:26:45.338221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.334542    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca" pod "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" (UID: "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:45.338221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.337420    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls" pod "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" (UID: "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:45.339593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.339552    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs" pod "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" (UID: "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:45.365672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.365638    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx" pod "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" (UID: "5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe"). InnerVolumeSpecName "kube-api-access-h9hwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:45.373802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.373775    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:26:45.476558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.447742    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.476558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.447777    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.476558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.447794    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hwx\" (UniqueName: \"kubernetes.io/projected/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-kube-api-access-h9hwx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.476558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.447810    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.549319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.549194    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\") pod \"8b69b063-32f7-4699-97cf-85a5e4a01f98\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") "
Oct 30 03:26:45.549319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.549308    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\" (UniqueName: \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\") pod \"8b69b063-32f7-4699-97cf-85a5e4a01f98\" (UID: \"8b69b063-32f7-4699-97cf-85a5e4a01f98\") "
Oct 30 03:26:45.554284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.550669    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume" pod "8b69b063-32f7-4699-97cf-85a5e4a01f98" (UID: "8b69b063-32f7-4699-97cf-85a5e4a01f98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:45.554284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.552903    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj" pod "8b69b063-32f7-4699-97cf-85a5e4a01f98" (UID: "8b69b063-32f7-4699-97cf-85a5e4a01f98"). InnerVolumeSpecName "kube-api-access-vr9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:45.651011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.650978    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b69b063-32f7-4699-97cf-85a5e4a01f98-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.651011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.651009    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9vj\" (UniqueName: \"kubernetes.io/projected/8b69b063-32f7-4699-97cf-85a5e4a01f98-kube-api-access-vr9vj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:26:45.965679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.965638    4214 generic.go:356] "Generic (PLEG): container finished" podID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" containerID="a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560" exitCode=0
Oct 30 03:26:45.966098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.965695    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"
Oct 30 03:26:45.966098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.965721    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" event={"ID":"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe","Type":"ContainerDied","Data":"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"}
Oct 30 03:26:45.966098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.965824    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md" event={"ID":"5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe","Type":"ContainerDied","Data":"cde6301c654233eeae317ac5df8d0111a43b211759a3662272bcfeda153c110b"}
Oct 30 03:26:45.966098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.965870    4214 scope.go:122] "RemoveContainer" containerID="a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"
Oct 30 03:26:45.970123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.970097    4214 generic.go:356] "Generic (PLEG): container finished" podID="8b69b063-32f7-4699-97cf-85a5e4a01f98" containerID="ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19" exitCode=0
Oct 30 03:26:45.970391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.970158    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"
Oct 30 03:26:45.970391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.970187    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" event={"ID":"8b69b063-32f7-4699-97cf-85a5e4a01f98","Type":"ContainerDied","Data":"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"}
Oct 30 03:26:45.970391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.970278    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w" event={"ID":"8b69b063-32f7-4699-97cf-85a5e4a01f98","Type":"ContainerDied","Data":"c90291007d5825f093d595f75b398c32c377d24fb910aa55d76abe11d677b618"}
Oct 30 03:26:45.988444 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.988418    4214 scope.go:122] "RemoveContainer" containerID="a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"
Oct 30 03:26:45.988839 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:45.988801    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560\": container with ID starting with a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560 not found: ID does not exist" containerID="a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"
Oct 30 03:26:45.988965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.988846    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560"} err="failed to get container status \"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560\": rpc error: code = NotFound desc = could not find container \"a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560\": container with ID starting with a9e91d32331a4b694aa007f51f37369d036ab0d0fc3d66776eb6894beb64d560 not found: ID does not exist"
Oct 30 03:26:45.988965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:45.988870    4214 scope.go:122] "RemoveContainer" containerID="ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"
Oct 30 03:26:46.002942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.002919    4214 scope.go:122] "RemoveContainer" containerID="ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"
Oct 30 03:26:46.003325 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:26:46.003301    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19\": container with ID starting with ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19 not found: ID does not exist" containerID="ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"
Oct 30 03:26:46.003438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.003334    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19"} err="failed to get container status \"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19\": rpc error: code = NotFound desc = could not find container \"ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19\": container with ID starting with ba47ee36ae724613988b2ef2f3824ea6c88f2cbb13c86bb9d6817215eaa24f19 not found: ID does not exist"
Oct 30 03:26:46.078007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.077975    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:26:46.083309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.083280    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5dbd868586-xww5w"]
Oct 30 03:26:46.128217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.128188    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:26:46.135103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.135076    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948965-7cb54b54b9-v24md"]
Oct 30 03:26:46.337638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:46.337604    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:26:47.070415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:47.070383    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" path="/var/lib/kubelet/pods/5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe/volumes"
Oct 30 03:26:47.070896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:47.070772    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b69b063-32f7-4699-97cf-85a5e4a01f98" path="/var/lib/kubelet/pods/8b69b063-32f7-4699-97cf-85a5e4a01f98/volumes"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.191278    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192686    4214 cpu_manager.go:414] "containerMap: removing container" podUID="53547826-3272-479b-8e84-e62933af4d27" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192717    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53547826-3272-479b-8e84-e62933af4d27" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192880    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192928    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192954    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192969    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192987    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8b69b063-32f7-4699-97cf-85a5e4a01f98" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.192995    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69b063-32f7-4699-97cf-85a5e4a01f98" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193014    4214 cpu_manager.go:414] "containerMap: removing container" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193022    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193034    4214 cpu_manager.go:414] "containerMap: removing container" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193042    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193059    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193066    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193085    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193094    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193108    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8736cfd8-158b-407c-a473-7755bbb10ce2" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193115    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8736cfd8-158b-407c-a473-7755bbb10ce2" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193128    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e19faf67-97c9-49d4-a80b-a32dc8c42204" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193136    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19faf67-97c9-49d4-a80b-a32dc8c42204" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193252    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193263    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193304    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193312    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193353    4214 cpu_manager.go:414] "containerMap: removing container" podUID="014aa7e5-12d6-49e4-b052-adc27bc226a4" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193361    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014aa7e5-12d6-49e4-b052-adc27bc226a4" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193550    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="014aa7e5-12d6-49e4-b052-adc27bc226a4" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193567    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="53547826-3272-479b-8e84-e62933af4d27" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193583    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7c1a97-08ef-4ea8-8d2b-8b8029ed7cfe" containerName="maas-api"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193591    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e829d47a-f503-4fd4-b798-cd88d7aa67d7" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193601    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69b063-32f7-4699-97cf-85a5e4a01f98" containerName="payload-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193612    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193621    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="df32909d-505e-4f7a-8afa-8ebfb3af15de" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193630    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193639    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83887e84-0d2e-46e9-b4e7-8e3cb5636160" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193657    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8736cfd8-158b-407c-a473-7755bbb10ce2" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193666    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19faf67-97c9-49d4-a80b-a32dc8c42204" containerName="payload-pre-processing"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193674    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.196253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.193687    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceb60bb-aae7-469c-a7f5-015f5a610c1f" containerName="istio-proxy"
Oct 30 03:26:51.198751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.197973    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.222886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.217524    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-055daa-openshift-default-dockercfg-bn5xs\""
Oct 30 03:26:51.254759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.252346    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332400    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332535    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmf6\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332569    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332665    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332702    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332758    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332807    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332867    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.334304 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.332910    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.434522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.434470    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.434727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.434579    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmf6\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.434727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.434605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.434976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.434950    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.434976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.434966    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435033    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435092    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435147    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435198    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435264    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435283    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435635    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435897 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435878    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.435956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.435905    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.437507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.437476    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.438720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.438693    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.459268 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.459162    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmf6\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.460876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.460824    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\") pod \"e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") " pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.531862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.531824    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:51.866282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.866252    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:26:51.869610 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:26:51.869574    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015c99fd_736c_4585_a94e_c20ecac4f1f4.slice/crio-0626aa2c995da85fa2f8c08ff28b556aec622317785539a87f7c2429c175ddc9 WatchSource:0}: Error finding container 0626aa2c995da85fa2f8c08ff28b556aec622317785539a87f7c2429c175ddc9: Status 404 returned error can't find the container with id 0626aa2c995da85fa2f8c08ff28b556aec622317785539a87f7c2429c175ddc9
Oct 30 03:26:51.872409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.872374    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:51.872523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.872451    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:51.872523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:51.872489    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:52.023154 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:52.023116    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerStarted","Data":"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"}
Oct 30 03:26:52.023394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:52.023179    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerStarted","Data":"0626aa2c995da85fa2f8c08ff28b556aec622317785539a87f7c2429c175ddc9"}
Oct 30 03:26:52.076013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:52.075958    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podStartSLOduration=1.075946382 podStartE2EDuration="1.075946382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:52.075814373 +0000 UTC m=+2751.149985167" watchObservedRunningTime="2026-10-30 03:26:52.075946382 +0000 UTC m=+2751.150117169"
Oct 30 03:26:52.532718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:52.532675    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:52.540191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:52.540162    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:53.032104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:53.032074    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:53.032938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:53.032915    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:26:55.885785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:55.883678    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:26:57.026134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.026101    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:26:57.029802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.029777    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.070767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.070735    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cf605a-openshift-default-dockercfg-dhxj6\""
Oct 30 03:26:57.085851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.085814    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155067    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155310    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155348    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155377    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155406    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155489    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmh7\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155521    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155549    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.156253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.155603    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.259599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.259558    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.260866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.260839    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.261411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.261389    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.261987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.261968    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.262456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.262438    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.263259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.263222    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmh7\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.263696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.263680    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.264853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.264834    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.265369    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.260615    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.262384    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265813 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.265785    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.262815    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.265866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.261913    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.267203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.267183    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.270972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.270939    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.287543 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.287496    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmh7\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.287993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.287978    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\") pod \"e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") " pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.345950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.345922    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:57.682198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.682010    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:26:57.696283 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.696241    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:57.696450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.696335    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:57.696450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:57.696376    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:26:58.096615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.096576    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerStarted","Data":"26b4d6f210fa446e37fc6b6134bbd5255589b044fec003e15eb02e8192361f3c"}
Oct 30 03:26:58.097068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.096638    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerStarted","Data":"2f9d73c0d4811e4e0b5ffa01cddf29a1a8f9ad75aa707594f20a5a10e4aae308"}
Oct 30 03:26:58.147251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.146679    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podStartSLOduration=2.14664596 podStartE2EDuration="2.14664596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:58.142983718 +0000 UTC m=+2757.217154512" watchObservedRunningTime="2026-10-30 03:26:58.14664596 +0000 UTC m=+2757.220816754"
Oct 30 03:26:58.346969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.346880    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:58.353258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.353207    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:58.515194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.515153    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:26:58.592753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.592724    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:26:58.593016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.592885    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.620614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.620576    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-055daa\""
Oct 30 03:26:58.633201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.633166    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-055daa-dockercfg-rscpp\""
Oct 30 03:26:58.726794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.726761    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.726794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.726796    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxx5c\" (UniqueName: \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.828340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.828304    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.828506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.828346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxx5c\" (UniqueName: \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.829025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.829004    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.862317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.857737    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxx5c\" (UniqueName: \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\") pod \"payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:58.942418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:58.942380    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:26:59.104502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:59.104473    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:59.105359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:59.105341    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:26:59.266617 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:26:59.266591    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8777733_f636_4161_915c_b01fd20df369.slice/crio-5b6ef92ea93ce68b83b0b7c3f9542c7255f3cfc6ba1d212de555acadd350f332 WatchSource:0}: Error finding container 5b6ef92ea93ce68b83b0b7c3f9542c7255f3cfc6ba1d212de555acadd350f332: Status 404 returned error can't find the container with id 5b6ef92ea93ce68b83b0b7c3f9542c7255f3cfc6ba1d212de555acadd350f332
Oct 30 03:26:59.268780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:26:59.268755    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:27:00.113940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:00.113888    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" event={"ID":"e8777733-f636-4161-915c-b01fd20df369","Type":"ContainerStarted","Data":"f192e7a638589547068e38d330378ed79a11ca093d199c20570d98be9143e7c3"}
Oct 30 03:27:00.114441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:00.113961    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" event={"ID":"e8777733-f636-4161-915c-b01fd20df369","Type":"ContainerStarted","Data":"5b6ef92ea93ce68b83b0b7c3f9542c7255f3cfc6ba1d212de555acadd350f332"}
Oct 30 03:27:00.114441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:00.114122    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:27:00.162025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:00.161978    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" podStartSLOduration=2.161964193 podStartE2EDuration="2.161964193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:00.159777322 +0000 UTC m=+2759.233948116" watchObservedRunningTime="2026-10-30 03:27:00.161964193 +0000 UTC m=+2759.236134987"
Oct 30 03:27:01.360836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.360792    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" podUID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" containerName="main" containerID="cri-o://561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5" gracePeriod=45
Oct 30 03:27:01.652353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.652327    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:27:01.678852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.678822    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.678904    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.678933    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.678959    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679009    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\" (UniqueName: \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679167    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\") pod \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\" (UID: \"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123\") "
Oct 30 03:27:01.679254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679179    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:01.679438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679246    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:01.680038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679569    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:01.680038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.679980    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:01.680038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.680002    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:01.680038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.680015    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:01.681673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.681635    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:01.681812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.681690    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:01.681812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.681797    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76" pod "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" (UID: "7d6374ea-3bbe-4b1b-bedb-9cdd397e3123"). InnerVolumeSpecName "kube-api-access-hzz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:01.781824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.781782    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:01.782019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.781845    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzz76\" (UniqueName: \"kubernetes.io/projected/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-kube-api-access-hzz76\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:01.782019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:01.781858    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.132861 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.132820    4214 generic.go:356] "Generic (PLEG): container finished" podID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" containerID="561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5" exitCode=0
Oct 30 03:27:02.133082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.132867    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" event={"ID":"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123","Type":"ContainerDied","Data":"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"}
Oct 30 03:27:02.133082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.132887    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"
Oct 30 03:27:02.133082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.132900    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw" event={"ID":"7d6374ea-3bbe-4b1b-bedb-9cdd397e3123","Type":"ContainerDied","Data":"182f013bd029aed0a66a90d1a0a24b643fb0db66522d75631cb1339d8cc66ebf"}
Oct 30 03:27:02.133082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.132919    4214 scope.go:122] "RemoveContainer" containerID="561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"
Oct 30 03:27:02.145048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.145024    4214 scope.go:122] "RemoveContainer" containerID="561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"
Oct 30 03:27:02.145416 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:02.145388    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5\": container with ID starting with 561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5 not found: ID does not exist" containerID="561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"
Oct 30 03:27:02.145521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.145429    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5"} err="failed to get container status \"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5\": rpc error: code = NotFound desc = could not find container \"561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5\": container with ID starting with 561e243e93f0b7961e4967bc5e3715422f44fa2a551547d59e6992be9577e5a5 not found: ID does not exist"
Oct 30 03:27:02.191573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.191535    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:27:02.204503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:02.204474    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-d823d1/auth-test-model-d823d1-kserve-6f8cbfccf7-b97nw"]
Oct 30 03:27:03.069636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:03.069603    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" path="/var/lib/kubelet/pods/7d6374ea-3bbe-4b1b-bedb-9cdd397e3123/volumes"
Oct 30 03:27:04.255070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:04.255028    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:27:07.117621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:07.117589    4214 scope.go:122] "RemoveContainer" containerID="f87b2aa220743c7c8c1ca79d454b30ba453fa492c47a9606c3a59e8eed7f88f6"
Oct 30 03:27:07.128823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:07.128801    4214 scope.go:122] "RemoveContainer" containerID="3bdf5a054ec8ed29915ff667962f44e8d9945429a50917f2ed81035bf239b998"
Oct 30 03:27:07.146223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:07.146140    4214 scope.go:122] "RemoveContainer" containerID="99525bf0eadfab529543ebc9899d701cceca00c5e008d11a1af817091a38918a"
Oct 30 03:27:09.165277 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:09.165225    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:27:09.255557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:09.255513    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing" containerID="cri-o://82a97d22a0ededc1046251673016455ca32eec91f8ef208849b96efb8f027564" gracePeriod=25
Oct 30 03:27:09.683911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:09.683868    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:10.909951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:10.909907    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" podUID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" containerName="main" containerID="cri-o://3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092" gracePeriod=45
Oct 30 03:27:11.127044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.127007    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:27:11.202987 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.202958    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:27:11.229383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.229348    4214 generic.go:356] "Generic (PLEG): container finished" podID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" containerID="3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092" exitCode=0
Oct 30 03:27:11.229553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.229406    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"
Oct 30 03:27:11.229553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.229427    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" event={"ID":"710722ea-d6f2-4c7e-844b-1ec0ef712f81","Type":"ContainerDied","Data":"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"}
Oct 30 03:27:11.229553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.229475    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8" event={"ID":"710722ea-d6f2-4c7e-844b-1ec0ef712f81","Type":"ContainerDied","Data":"6ce5e4c6823b9be46a4a57c9e5e16406fb9f2d160a753aebb703b82f4043bc7a"}
Oct 30 03:27:11.229553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.229498    4214 scope.go:122] "RemoveContainer" containerID="3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"
Oct 30 03:27:11.242021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.241990    4214 scope.go:122] "RemoveContainer" containerID="3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"
Oct 30 03:27:11.242377 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:11.242346    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092\": container with ID starting with 3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092 not found: ID does not exist" containerID="3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"
Oct 30 03:27:11.242471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.242386    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092"} err="failed to get container status \"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092\": rpc error: code = NotFound desc = could not find container \"3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092\": container with ID starting with 3b6b2128a56e8f58af7e6b84a932012517569628426eb143e3e848aec6401092 not found: ID does not exist"
Oct 30 03:27:11.285544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285514    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.285755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285574    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.285755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285654    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.285755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285725    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.285755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285752    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\" (UniqueName: \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.286008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285828    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:11.286008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.285940    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\" (UniqueName: \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\") pod \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\" (UID: \"710722ea-d6f2-4c7e-844b-1ec0ef712f81\") "
Oct 30 03:27:11.286122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.286005    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:11.286209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.286177    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:11.286692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.286666    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.286692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.286693    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.286866 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.286706    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.288798 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.288747    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:11.288908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.288837    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:11.290405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.290378    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk" pod "710722ea-d6f2-4c7e-844b-1ec0ef712f81" (UID: "710722ea-d6f2-4c7e-844b-1ec0ef712f81"). InnerVolumeSpecName "kube-api-access-nhxxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:11.402276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.400264    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710722ea-d6f2-4c7e-844b-1ec0ef712f81-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.402276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.400308    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxxk\" (UniqueName: \"kubernetes.io/projected/710722ea-d6f2-4c7e-844b-1ec0ef712f81-kube-api-access-nhxxk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.402276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.400324    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710722ea-d6f2-4c7e-844b-1ec0ef712f81-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:11.580590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.580552    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:27:11.593090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:11.593061    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-874ab5/auth-test-model-874ab5-kserve-c984bdf9c-qm7v8"]
Oct 30 03:27:13.070188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:13.070148    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" path="/var/lib/kubelet/pods/710722ea-d6f2-4c7e-844b-1ec0ef712f81/volumes"
Oct 30 03:27:18.468688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:18.468653    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:27:18.469538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:18.469427    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy" containerID="cri-o://71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee" gracePeriod=30
Oct 30 03:27:19.682146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:19.682103    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:21.992157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992126    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:27:21.992867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992852    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" containerName="main"
Oct 30 03:27:21.992918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992869    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" containerName="main"
Oct 30 03:27:21.992918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992885    4214 cpu_manager.go:414] "containerMap: removing container" podUID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" containerName="main"
Oct 30 03:27:21.992918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992890    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" containerName="main"
Oct 30 03:27:21.993020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.992998    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6374ea-3bbe-4b1b-bedb-9cdd397e3123" containerName="main"
Oct 30 03:27:21.993020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.993008    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="710722ea-d6f2-4c7e-844b-1ec0ef712f81" containerName="main"
Oct 30 03:27:21.997831 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:21.997813    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.017216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.017182    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:27:22.025157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.025129    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-055daa-serving-cert\""
Oct 30 03:27:22.061982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.061951    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.062161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.062077    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z264\" (UniqueName: \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.062161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.062151    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.062325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.062286    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.163489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.163449    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.163674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.163555    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.163674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.163615    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z264\" (UniqueName: \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.163674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.163661    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.164457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.164428    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.166299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.166273    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.166652 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.166628    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.184083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.184052    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z264\" (UniqueName: \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\") pod \"maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.313474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.313441    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:22.627371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:22.627340    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:27:22.627631 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:27:22.627611    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d93fa35_c0e3_49f4_8f7d_adc83dc710a1.slice/crio-3e4c1ac1bb2ef154237eebce08a1fe36325416c0a76d9bd20e479295c597f592 WatchSource:0}: Error finding container 3e4c1ac1bb2ef154237eebce08a1fe36325416c0a76d9bd20e479295c597f592: Status 404 returned error can't find the container with id 3e4c1ac1bb2ef154237eebce08a1fe36325416c0a76d9bd20e479295c597f592
Oct 30 03:27:23.395840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.395802    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" event={"ID":"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1","Type":"ContainerStarted","Data":"4ba3e23b5db557735c2162b95d5114e8d526f4e2e2bc906f0f8479bd4ecb49aa"}
Oct 30 03:27:23.396295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.395865    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" event={"ID":"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1","Type":"ContainerStarted","Data":"3e4c1ac1bb2ef154237eebce08a1fe36325416c0a76d9bd20e479295c597f592"}
Oct 30 03:27:23.396295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.396043    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:23.794024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.793996    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:27:23.844209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.844143    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" podStartSLOduration=2.434726705 podStartE2EDuration="2.844123038s" totalImagesPullingTime="409.396333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:23.454744898 +0000 UTC m=+2782.528915694" watchObservedRunningTime="2026-10-30 03:27:23.844123038 +0000 UTC m=+2782.918293837"
Oct 30 03:27:23.892537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892499    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892574    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892606    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892645    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892674    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892769    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892822    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892866    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892913    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\") pod \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\" (UID: \"80ce25d3-ead3-45dc-b5d4-fa157d141a01\") "
Oct 30 03:27:23.892992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.892954    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:23.893429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.893407    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.893718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.893692    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:23.893839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.893707    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:23.893953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.893929    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:23.894061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.893940    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:23.896479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.896438    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:23.896600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.896549    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "kube-api-access-8dbt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:23.901665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.901637    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:23.901763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.901649    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo" pod "80ce25d3-ead3-45dc-b5d4-fa157d141a01" (UID: "80ce25d3-ead3-45dc-b5d4-fa157d141a01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:23.917284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.917252    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994170    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbt9\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-kube-api-access-8dbt9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994201    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994211    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994220    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994239    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994246    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994254    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:23.994267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:23.994261    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce25d3-ead3-45dc-b5d4-fa157d141a01-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:24.430907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.430876    4214 generic.go:356] "Generic (PLEG): container finished" podID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerID="71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee" exitCode=0
Oct 30 03:27:24.431363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.430943    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"
Oct 30 03:27:24.431363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.430960    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerDied","Data":"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"}
Oct 30 03:27:24.431363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.431027    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx" event={"ID":"80ce25d3-ead3-45dc-b5d4-fa157d141a01","Type":"ContainerDied","Data":"9c70c74bb517d8dc8621d89c3b2347b56d3a6865df0cd10db5ff5625eaa413af"}
Oct 30 03:27:24.431363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.431059    4214 scope.go:122] "RemoveContainer" containerID="71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"
Oct 30 03:27:24.445200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.445177    4214 scope.go:122] "RemoveContainer" containerID="cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"
Oct 30 03:27:24.457742 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.457719    4214 scope.go:122] "RemoveContainer" containerID="71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"
Oct 30 03:27:24.458090 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:24.458060    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee\": container with ID starting with 71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee not found: ID does not exist" containerID="71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"
Oct 30 03:27:24.458185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.458102    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee"} err="failed to get container status \"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee\": rpc error: code = NotFound desc = could not find container \"71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee\": container with ID starting with 71f01a5bea0868435101f149425159b6ff7c9624a10bd00eb608f287c65cdcee not found: ID does not exist"
Oct 30 03:27:24.458185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.458128    4214 scope.go:122] "RemoveContainer" containerID="cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"
Oct 30 03:27:24.458439 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:24.458417    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab\": container with ID starting with cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab not found: ID does not exist" containerID="cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"
Oct 30 03:27:24.458500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.458446    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab"} err="failed to get container status \"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab\": rpc error: code = NotFound desc = could not find container \"cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab\": container with ID starting with cbc9041c1f1f6606c28a7ed6754be53e3d85c24faabd44530289cd826d05efab not found: ID does not exist"
Oct 30 03:27:24.487537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.487502    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:27:24.491351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:24.491319    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e2727-openshift-default-9889b484f-jn4jx"]
Oct 30 03:27:25.071284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:25.071248    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" path="/var/lib/kubelet/pods/80ce25d3-ead3-45dc-b5d4-fa157d141a01/volumes"
Oct 30 03:27:27.249978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.249940    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:27:27.250843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.250819    4214 cpu_manager.go:414] "containerMap: removing container" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:27:27.250968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.250844    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:27:27.251089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.251073    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:27:27.256569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.256545    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.274818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.274790    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:27:27.282163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.282101    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cf605a-serving-cert\""
Oct 30 03:27:27.362486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.362449    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sslv\" (UniqueName: \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.362676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.362533    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.362727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.362667    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.362762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.362746    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.464610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.464563    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sslv\" (UniqueName: \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.464803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.464658    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.464803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.464749    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.464960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.464803    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.466619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.466585    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.468459 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.468427    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.469418 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.469395    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.487271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.487202    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sslv\" (UniqueName: \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\") pod \"maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.577062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.577032    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:27.888885 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:27:27.888848    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a152e51_e5bf_426d_9e44_4b4c862971e1.slice/crio-0b8c0544d8ff39a10d528bd7163039099a8aa843a3f7deb408a8151459e40605 WatchSource:0}: Error finding container 0b8c0544d8ff39a10d528bd7163039099a8aa843a3f7deb408a8151459e40605: Status 404 returned error can't find the container with id 0b8c0544d8ff39a10d528bd7163039099a8aa843a3f7deb408a8151459e40605
Oct 30 03:27:27.889277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:27.889237    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:27:28.487729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:28.487638    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" event={"ID":"1a152e51-e5bf-426d-9e44-4b4c862971e1","Type":"ContainerStarted","Data":"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"}
Oct 30 03:27:28.487729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:28.487703    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" event={"ID":"1a152e51-e5bf-426d-9e44-4b4c862971e1","Type":"ContainerStarted","Data":"0b8c0544d8ff39a10d528bd7163039099a8aa843a3f7deb408a8151459e40605"}
Oct 30 03:27:28.488317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:28.487749    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:28.532704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:28.532654    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" podStartSLOduration=1.190212883 podStartE2EDuration="1.532641315s" totalImagesPullingTime="342.428432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:28.528385388 +0000 UTC m=+2787.602556183" watchObservedRunningTime="2026-10-30 03:27:28.532641315 +0000 UTC m=+2787.606812141"
Oct 30 03:27:29.442309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.441850    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:27:29.500903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.500823    4214 generic.go:356] "Generic (PLEG): container finished" podID="630f8512-3070-42d8-8bb3-fea98031b56c" containerID="82a97d22a0ededc1046251673016455ca32eec91f8ef208849b96efb8f027564" exitCode=0
Oct 30 03:27:29.501478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.500871    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" event={"ID":"630f8512-3070-42d8-8bb3-fea98031b56c","Type":"ContainerDied","Data":"82a97d22a0ededc1046251673016455ca32eec91f8ef208849b96efb8f027564"}
Oct 30 03:27:29.557673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.557641    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:27:29.649293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.649244    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 03:27:29.694115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.694076    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\") pod \"630f8512-3070-42d8-8bb3-fea98031b56c\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") "
Oct 30 03:27:29.694335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.694148    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\" (UniqueName: \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\") pod \"630f8512-3070-42d8-8bb3-fea98031b56c\" (UID: \"630f8512-3070-42d8-8bb3-fea98031b56c\") "
Oct 30 03:27:29.694951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.694923    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume" pod "630f8512-3070-42d8-8bb3-fea98031b56c" (UID: "630f8512-3070-42d8-8bb3-fea98031b56c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:29.703803 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.703747    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4" pod "630f8512-3070-42d8-8bb3-fea98031b56c" (UID: "630f8512-3070-42d8-8bb3-fea98031b56c"). InnerVolumeSpecName "kube-api-access-vwtm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:29.796520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.796492    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630f8512-3070-42d8-8bb3-fea98031b56c-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:29.796520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:29.796521    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwtm4\" (UniqueName: \"kubernetes.io/projected/630f8512-3070-42d8-8bb3-fea98031b56c-kube-api-access-vwtm4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:30.511835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:30.511803    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"
Oct 30 03:27:30.512340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:30.511802    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5" event={"ID":"630f8512-3070-42d8-8bb3-fea98031b56c","Type":"ContainerDied","Data":"b0ad4e1727f5e88cbbcc69549db3de2b051e91df91653d0e739923da2b2f71fb"}
Oct 30 03:27:30.512340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:30.511940    4214 scope.go:122] "RemoveContainer" containerID="82a97d22a0ededc1046251673016455ca32eec91f8ef208849b96efb8f027564"
Oct 30 03:27:30.566876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:30.566715    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:27:30.579301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:30.579272    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac7936-6dbb48c479-zkvg5"]
Oct 30 03:27:31.080923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:31.080884    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" path="/var/lib/kubelet/pods/630f8512-3070-42d8-8bb3-fea98031b56c/volumes"
Oct 30 03:27:34.509016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:34.508986    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:27:34.650059 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:34.649998    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing" containerID="cri-o://c86ac29a4220eeb133f4054dc02eea42271b084aa4f07230442fb4b498f6bb6f" gracePeriod=25
Oct 30 03:27:38.940989 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:38.940913    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main" containerID="cri-o://be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e" gracePeriod=45
Oct 30 03:27:39.178769 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:39.178724    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:27:39.216694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.216659    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:27:39.327613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.327212    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.328196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.327692    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.328196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.328155    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.328385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.328276    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.328539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.328517    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.328684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.328665    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\" (UniqueName: \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.329049 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.329028    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.330907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.329315    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.330907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.329546    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\" (UniqueName: \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\") pod \"bde07f9d-958a-4478-9c16-08d908902b66\" (UID: \"bde07f9d-958a-4478-9c16-08d908902b66\") "
Oct 30 03:27:39.330907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.330221    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.330907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.330259    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.330907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.330272    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.337265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.336853    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:39.337265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.336889    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.337265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.337035    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8" pod "bde07f9d-958a-4478-9c16-08d908902b66" (UID: "bde07f9d-958a-4478-9c16-08d908902b66"). InnerVolumeSpecName "kube-api-access-v94v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.431657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.431620    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94v8\" (UniqueName: \"kubernetes.io/projected/bde07f9d-958a-4478-9c16-08d908902b66-kube-api-access-v94v8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.431657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.431650    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bde07f9d-958a-4478-9c16-08d908902b66-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.431657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.431659    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bde07f9d-958a-4478-9c16-08d908902b66-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.580974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.580940    4214 generic.go:356] "Generic (PLEG): container finished" podID="bde07f9d-958a-4478-9c16-08d908902b66" containerID="be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e" exitCode=0
Oct 30 03:27:39.581204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.580987    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"
Oct 30 03:27:39.581204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.581016    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" event={"ID":"bde07f9d-958a-4478-9c16-08d908902b66","Type":"ContainerDied","Data":"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"}
Oct 30 03:27:39.581204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.581052    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh" event={"ID":"bde07f9d-958a-4478-9c16-08d908902b66","Type":"ContainerDied","Data":"aa4729d59c3648ae622246f7460d613d091034ab8f18d1629732e21f7786d741"}
Oct 30 03:27:39.581204 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.581074    4214 scope.go:122] "RemoveContainer" containerID="be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"
Oct 30 03:27:39.593217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.593190    4214 scope.go:122] "RemoveContainer" containerID="be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"
Oct 30 03:27:39.593514 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:39.593494    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e\": container with ID starting with be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e not found: ID does not exist" containerID="be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"
Oct 30 03:27:39.593571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.593521    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e"} err="failed to get container status \"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e\": rpc error: code = NotFound desc = could not find container \"be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e\": container with ID starting with be1f4355afd42fda72e8a36e64ab93814110a4566a99fcfc50703eb96b8f760e not found: ID does not exist"
Oct 30 03:27:39.635590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.635560    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 03:27:39.641978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:39.641951    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/premium-w3-d9f3ac-kserve-849b7f5899-mv8zh"]
Oct 30 03:27:41.068017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:41.067983    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde07f9d-958a-4478-9c16-08d908902b66" path="/var/lib/kubelet/pods/bde07f9d-958a-4478-9c16-08d908902b66/volumes"
Oct 30 03:27:44.494116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:44.494077    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:50.619745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:50.619701    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:27:53.505628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:53.505593    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:27:53.506357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:53.506327    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy" containerID="cri-o://98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4" gracePeriod=30
Oct 30 03:27:54.497256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:54.496943    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:54.788775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:54.788414    4214 generic.go:356] "Generic (PLEG): container finished" podID="d18381b7-5795-4c53-954c-33859dee342b" containerID="c86ac29a4220eeb133f4054dc02eea42271b084aa4f07230442fb4b498f6bb6f" exitCode=0
Oct 30 03:27:54.788775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:54.788705    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" event={"ID":"d18381b7-5795-4c53-954c-33859dee342b","Type":"ContainerDied","Data":"c86ac29a4220eeb133f4054dc02eea42271b084aa4f07230442fb4b498f6bb6f"}
Oct 30 03:27:54.909869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:54.909843    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 03:27:55.071121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.067575    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\" (UniqueName: \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\") pod \"d18381b7-5795-4c53-954c-33859dee342b\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") "
Oct 30 03:27:55.071121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.067835    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\") pod \"d18381b7-5795-4c53-954c-33859dee342b\" (UID: \"d18381b7-5795-4c53-954c-33859dee342b\") "
Oct 30 03:27:55.071121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.068838    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume" pod "d18381b7-5795-4c53-954c-33859dee342b" (UID: "d18381b7-5795-4c53-954c-33859dee342b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:55.071703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.071674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq" pod "d18381b7-5795-4c53-954c-33859dee342b" (UID: "d18381b7-5795-4c53-954c-33859dee342b"). InnerVolumeSpecName "kube-api-access-wzzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:55.171256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.170181    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18381b7-5795-4c53-954c-33859dee342b-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:55.171256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.170217    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzjq\" (UniqueName: \"kubernetes.io/projected/d18381b7-5795-4c53-954c-33859dee342b-kube-api-access-wzzjq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:55.620637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.620593    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" podUID="e8777733-f636-4161-915c-b01fd20df369" containerName="payload-processing" containerID="cri-o://f192e7a638589547068e38d330378ed79a11ca093d199c20570d98be9143e7c3" gracePeriod=25
Oct 30 03:27:55.801396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.801356    4214 generic.go:356] "Generic (PLEG): container finished" podID="e8777733-f636-4161-915c-b01fd20df369" containerID="f192e7a638589547068e38d330378ed79a11ca093d199c20570d98be9143e7c3" exitCode=0
Oct 30 03:27:55.801825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.801407    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" event={"ID":"e8777733-f636-4161-915c-b01fd20df369","Type":"ContainerDied","Data":"f192e7a638589547068e38d330378ed79a11ca093d199c20570d98be9143e7c3"}
Oct 30 03:27:55.803817 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.803761    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q" event={"ID":"d18381b7-5795-4c53-954c-33859dee342b","Type":"ContainerDied","Data":"525fcf557522f1a810c2dce5b5150a1fd1fe2a9986ec4983c269a07b151724d0"}
Oct 30 03:27:55.804095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.803829    4214 scope.go:122] "RemoveContainer" containerID="c86ac29a4220eeb133f4054dc02eea42271b084aa4f07230442fb4b498f6bb6f"
Oct 30 03:27:55.804095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.804003    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"
Oct 30 03:27:55.869584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.869538    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 03:27:55.880113 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:55.880023    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d9f3ac-b564b45c9-78h5q"]
Oct 30 03:27:56.081039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.081011    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:27:56.108945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.108908    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\" (UniqueName: \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\") pod \"e8777733-f636-4161-915c-b01fd20df369\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") "
Oct 30 03:27:56.109139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.108983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\") pod \"e8777733-f636-4161-915c-b01fd20df369\" (UID: \"e8777733-f636-4161-915c-b01fd20df369\") "
Oct 30 03:27:56.109540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.109396    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume" pod "e8777733-f636-4161-915c-b01fd20df369" (UID: "e8777733-f636-4161-915c-b01fd20df369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:56.109540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.109511    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8777733-f636-4161-915c-b01fd20df369-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.111066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.111046    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c" pod "e8777733-f636-4161-915c-b01fd20df369" (UID: "e8777733-f636-4161-915c-b01fd20df369"). InnerVolumeSpecName "kube-api-access-mxx5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:56.210853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.210758    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxx5c\" (UniqueName: \"kubernetes.io/projected/e8777733-f636-4161-915c-b01fd20df369-kube-api-access-mxx5c\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.815730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.815549    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v" event={"ID":"e8777733-f636-4161-915c-b01fd20df369","Type":"ContainerDied","Data":"5b6ef92ea93ce68b83b0b7c3f9542c7255f3cfc6ba1d212de555acadd350f332"}
Oct 30 03:27:56.815730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.815636    4214 scope.go:122] "RemoveContainer" containerID="f192e7a638589547068e38d330378ed79a11ca093d199c20570d98be9143e7c3"
Oct 30 03:27:56.817891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.817873    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"
Oct 30 03:27:56.901305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.901200    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:27:56.907158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:56.907118    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-5f8b69cb54-5hk6v"]
Oct 30 03:27:57.081378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:57.080242    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18381b7-5795-4c53-954c-33859dee342b" path="/var/lib/kubelet/pods/d18381b7-5795-4c53-954c-33859dee342b/volumes"
Oct 30 03:27:57.081378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:57.080968    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8777733-f636-4161-915c-b01fd20df369" path="/var/lib/kubelet/pods/e8777733-f636-4161-915c-b01fd20df369/volumes"
Oct 30 03:27:58.824502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.824349    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:27:58.843161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.842908    4214 generic.go:356] "Generic (PLEG): container finished" podID="44c4af4c-7c45-440f-88b8-066a5020071d" containerID="98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4" exitCode=0
Oct 30 03:27:58.843161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.842957    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"
Oct 30 03:27:58.843161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.842956    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" event={"ID":"44c4af4c-7c45-440f-88b8-066a5020071d","Type":"ContainerDied","Data":"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"}
Oct 30 03:27:58.843161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.843032    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh" event={"ID":"44c4af4c-7c45-440f-88b8-066a5020071d","Type":"ContainerDied","Data":"01bf6fa696dbf948c106e2aa7fa33a7e5be5d8276d890654388e211a72a9c745"}
Oct 30 03:27:58.843161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.843056    4214 scope.go:122] "RemoveContainer" containerID="98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"
Oct 30 03:27:58.855439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.855380    4214 scope.go:122] "RemoveContainer" containerID="98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"
Oct 30 03:27:58.855765 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:27:58.855738    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4\": container with ID starting with 98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4 not found: ID does not exist" containerID="98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"
Oct 30 03:27:58.855874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.855777    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4"} err="failed to get container status \"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4\": rpc error: code = NotFound desc = could not find container \"98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4\": container with ID starting with 98988ad7de235d9029c5b4b560e2e53446cd247cc5a86eb69d5307ed43fa38a4 not found: ID does not exist"
Oct 30 03:27:58.977570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977512    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.977809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977648    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.977809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977689    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.977809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977736    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.977809 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977788    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.978030 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.977913    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.978088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.978043    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.978088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.978072    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.978296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.978107    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\") pod \"44c4af4c-7c45-440f-88b8-066a5020071d\" (UID: \"44c4af4c-7c45-440f-88b8-066a5020071d\") "
Oct 30 03:27:58.980402 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.980085    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.980719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.980674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.981015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.980984    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:58.984301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.983334    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.984301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.983578    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.984301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.984207    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.984755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.984720    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:58.985016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.984990    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:58.985671 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:58.985646    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn" pod "44c4af4c-7c45-440f-88b8-066a5020071d" (UID: "44c4af4c-7c45-440f-88b8-066a5020071d"). InnerVolumeSpecName "kube-api-access-c5msn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:59.039261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.039123    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:27:59.079713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079681    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.079713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079709    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.079713 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079719    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c4af4c-7c45-440f-88b8-066a5020071d-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079729    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079737    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079744    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079753    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c4af4c-7c45-440f-88b8-066a5020071d-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079763    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5msn\" (UniqueName: \"kubernetes.io/projected/44c4af4c-7c45-440f-88b8-066a5020071d-kube-api-access-c5msn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.080035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.079772    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c4af4c-7c45-440f-88b8-066a5020071d-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:27:59.198172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.196015    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:27:59.200060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:27:59.200033    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-jktlh"]
Oct 30 03:28:00.146455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.146407    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:28:00.148381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.146482    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:28:00.865050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.865016    4214 generic.go:356] "Generic (PLEG): container finished" podID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerID="26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca" exitCode=0
Oct 30 03:28:00.865288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.865066    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerDied","Data":"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"}
Oct 30 03:28:00.865492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.865472    4214 scope.go:122] "RemoveContainer" containerID="26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"
Oct 30 03:28:00.867198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.867148    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:00.867340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.867264    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:00.867340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:00.867317    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:01.072767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:01.072723    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" path="/var/lib/kubelet/pods/44c4af4c-7c45-440f-88b8-066a5020071d/volumes"
Oct 30 03:28:01.875570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:01.875530    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerStarted","Data":"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"}
Oct 30 03:28:02.532860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:02.532824    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:28:02.533072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:02.532982    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:02.533072 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:02.533031    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:03.533038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.533002    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:03.534583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.533062    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:03.894592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.894384    4214 generic.go:356] "Generic (PLEG): container finished" podID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerID="26b4d6f210fa446e37fc6b6134bbd5255589b044fec003e15eb02e8192361f3c" exitCode=0
Oct 30 03:28:03.894592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.894459    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerDied","Data":"26b4d6f210fa446e37fc6b6134bbd5255589b044fec003e15eb02e8192361f3c"}
Oct 30 03:28:03.894876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.894862    4214 scope.go:122] "RemoveContainer" containerID="26b4d6f210fa446e37fc6b6134bbd5255589b044fec003e15eb02e8192361f3c"
Oct 30 03:28:03.896317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.896289    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:03.896427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.896352    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:03.896427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:03.896388    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:28:04.532859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:04.532824    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:04.533069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:04.532876    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:04.770569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:04.770526    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 03:28:04.802938 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:04.802902    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 03:28:04.909976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:04.909940    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerStarted","Data":"6dafea288556d14c4ba27b8f87a3bd64ec9ac323291c0127f23b64cebba942f1"}
Oct 30 03:28:05.352778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:05.352399    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:28:05.353209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:05.353187    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:05.353438 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:05.353387    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:05.533255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:05.532758    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:05.533255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:05.532815    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:06.355254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:06.347952    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:06.355254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:06.348016    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:06.532875 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:06.532843    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:06.533115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:06.533090    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:07.353252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:07.349662    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:07.353252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:07.349728    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:07.532474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:07.532392    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Oct 30 03:28:07.532474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:07.532449    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Oct 30 03:28:08.032350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.032318    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:28:08.089148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.089115    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:28:08.380381 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.380344    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:08.380971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.380935    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:08.553842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.553503    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:28:08.977248 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:08.976882    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:28:09.204792 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:09.204756    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:28:09.353668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.353622    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:09.353904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.353687    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:09.771275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.771142    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" podUID="2618539f-c69f-43ac-8db3-30704cfaf7cb" containerName="payload-processing" containerID="cri-o://62646af75d121d0b6a54dae1b79a1f92671de95c8c960fbf2ec741631937eb34" gracePeriod=25
Oct 30 03:28:09.803180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.803143    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing" containerID="cri-o://b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f" gracePeriod=25
Oct 30 03:28:09.988078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.987157    4214 generic.go:356] "Generic (PLEG): container finished" podID="2618539f-c69f-43ac-8db3-30704cfaf7cb" containerID="62646af75d121d0b6a54dae1b79a1f92671de95c8c960fbf2ec741631937eb34" exitCode=0
Oct 30 03:28:09.988078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:09.987251    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" event={"ID":"2618539f-c69f-43ac-8db3-30704cfaf7cb","Type":"ContainerDied","Data":"62646af75d121d0b6a54dae1b79a1f92671de95c8c960fbf2ec741631937eb34"}
Oct 30 03:28:10.108066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.107178    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 03:28:10.172781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.172747    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\") pod \"2618539f-c69f-43ac-8db3-30704cfaf7cb\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") "
Oct 30 03:28:10.172980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.172865    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\" (UniqueName: \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\") pod \"2618539f-c69f-43ac-8db3-30704cfaf7cb\" (UID: \"2618539f-c69f-43ac-8db3-30704cfaf7cb\") "
Oct 30 03:28:10.173217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.173191    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume" pod "2618539f-c69f-43ac-8db3-30704cfaf7cb" (UID: "2618539f-c69f-43ac-8db3-30704cfaf7cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:10.173467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.173450    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2618539f-c69f-43ac-8db3-30704cfaf7cb-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:10.175080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.175051    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x" pod "2618539f-c69f-43ac-8db3-30704cfaf7cb" (UID: "2618539f-c69f-43ac-8db3-30704cfaf7cb"). InnerVolumeSpecName "kube-api-access-v9h6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:10.296310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.296270    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9h6x\" (UniqueName: \"kubernetes.io/projected/2618539f-c69f-43ac-8db3-30704cfaf7cb-kube-api-access-v9h6x\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:10.347761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.347662    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 03:28:10.348069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.348034    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 03:28:10.996585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.996553    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"
Oct 30 03:28:10.997054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.996581    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc" event={"ID":"2618539f-c69f-43ac-8db3-30704cfaf7cb","Type":"ContainerDied","Data":"ba2b5a5c47e02e3b5aa813fdf1af6d6bb9e10bb828229b18931336eb67c49692"}
Oct 30 03:28:10.997054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:10.996647    4214 scope.go:122] "RemoveContainer" containerID="62646af75d121d0b6a54dae1b79a1f92671de95c8c960fbf2ec741631937eb34"
Oct 30 03:28:11.061431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:11.054016    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 03:28:11.080735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:11.080707    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e11f03-855bbc8d7b-t87kc"]
Oct 30 03:28:11.351402 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:11.351368    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:28:11.351787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:11.351765    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:28:11.352698 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:11.352669    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:28:13.069344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:13.069309    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2618539f-c69f-43ac-8db3-30704cfaf7cb" path="/var/lib/kubelet/pods/2618539f-c69f-43ac-8db3-30704cfaf7cb/volumes"
Oct 30 03:28:13.089845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:13.089807    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing" containerID="cri-o://146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e" gracePeriod=25
Oct 30 03:28:14.068462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.068379    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main" containerID="cri-o://0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1" gracePeriod=45
Oct 30 03:28:14.430419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.429940    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556171    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\" (UniqueName: \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556305    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556427    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556593    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556631    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.556669    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\") pod \"fcc5f5b9-e941-4236-aae5-3c4059863209\" (UID: \"fcc5f5b9-e941-4236-aae5-3c4059863209\") "
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.558676    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.558953    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.560900    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.562189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.561861    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.564519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.564450    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:14.565478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.565438    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws" pod "fcc5f5b9-e941-4236-aae5-3c4059863209" (UID: "fcc5f5b9-e941-4236-aae5-3c4059863209"). InnerVolumeSpecName "kube-api-access-zp2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658787    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658828    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658843    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658857    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc5f5b9-e941-4236-aae5-3c4059863209-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658869    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc5f5b9-e941-4236-aae5-3c4059863209-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.658886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:14.658882    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp2ws\" (UniqueName: \"kubernetes.io/projected/fcc5f5b9-e941-4236-aae5-3c4059863209-kube-api-access-zp2ws\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:15.030911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.030827    4214 generic.go:356] "Generic (PLEG): container finished" podID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerID="0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1" exitCode=0
Oct 30 03:28:15.030911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.030871    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"
Oct 30 03:28:15.030911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.030888    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" event={"ID":"fcc5f5b9-e941-4236-aae5-3c4059863209","Type":"ContainerDied","Data":"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"}
Oct 30 03:28:15.031162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.030928    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6" event={"ID":"fcc5f5b9-e941-4236-aae5-3c4059863209","Type":"ContainerDied","Data":"10527c414122e6f42c8c951919016b2562f5d7e15f2e3ffb31abbe49b704ed03"}
Oct 30 03:28:15.031162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.030944    4214 scope.go:122] "RemoveContainer" containerID="0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"
Oct 30 03:28:15.043997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.043961    4214 scope.go:122] "RemoveContainer" containerID="0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"
Oct 30 03:28:15.044335 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:15.044312    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1\": container with ID starting with 0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1 not found: ID does not exist" containerID="0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"
Oct 30 03:28:15.044415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.044346    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1"} err="failed to get container status \"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1\": rpc error: code = NotFound desc = could not find container \"0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1\": container with ID starting with 0bbb28cc283476ca27f6f1cf7ae756f1a05071177c0725626465a14e969817a1 not found: ID does not exist"
Oct 30 03:28:15.090009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.089983    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:28:15.101735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:15.101713    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/distinct-2-w1-e11f03-kserve-7c54c57cf5-b6vk6"]
Oct 30 03:28:16.565467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:16.565418    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:17.068221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:17.068184    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" path="/var/lib/kubelet/pods/fcc5f5b9-e941-4236-aae5-3c4059863209/volumes"
Oct 30 03:28:19.572529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:19.572493    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:26.565311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:26.565267    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:27.033168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.033086    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:28:27.033612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.033583    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" podUID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" containerName="maas-api" containerID="cri-o://5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909" gracePeriod=30
Oct 30 03:28:27.606835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.606804    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:28:27.636493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.636463    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\") pod \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") "
Oct 30 03:28:27.637193 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.637169    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\" (UniqueName: \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\") pod \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") "
Oct 30 03:28:27.637788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.637767    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\") pod \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") "
Oct 30 03:28:27.637929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.637856    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\") pod \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\" (UID: \"66f45ba8-c10a-4a44-b87d-7f64ef6f1693\") "
Oct 30 03:28:27.638327 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.638299    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca" pod "66f45ba8-c10a-4a44-b87d-7f64ef6f1693" (UID: "66f45ba8-c10a-4a44-b87d-7f64ef6f1693"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:27.638583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.638563    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:27.640210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.640183    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs" pod "66f45ba8-c10a-4a44-b87d-7f64ef6f1693" (UID: "66f45ba8-c10a-4a44-b87d-7f64ef6f1693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:27.640349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.640221    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t" pod "66f45ba8-c10a-4a44-b87d-7f64ef6f1693" (UID: "66f45ba8-c10a-4a44-b87d-7f64ef6f1693"). InnerVolumeSpecName "kube-api-access-j627t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:27.641577 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.641555    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls" pod "66f45ba8-c10a-4a44-b87d-7f64ef6f1693" (UID: "66f45ba8-c10a-4a44-b87d-7f64ef6f1693"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:27.739387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.739353    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:27.739387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.739385    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j627t\" (UniqueName: \"kubernetes.io/projected/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-kube-api-access-j627t\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:27.739387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:27.739396    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66f45ba8-c10a-4a44-b87d-7f64ef6f1693-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:28.140823 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.140785    4214 generic.go:356] "Generic (PLEG): container finished" podID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" containerID="5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909" exitCode=0
Oct 30 03:28:28.141031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.140841    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"
Oct 30 03:28:28.141031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.140848    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" event={"ID":"66f45ba8-c10a-4a44-b87d-7f64ef6f1693","Type":"ContainerDied","Data":"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"}
Oct 30 03:28:28.141031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.140919    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q" event={"ID":"66f45ba8-c10a-4a44-b87d-7f64ef6f1693","Type":"ContainerDied","Data":"99ed7f3032dcc37d56f896893658ffeb4324a8b4e22235488a6773e05d9476d5"}
Oct 30 03:28:28.141031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.140948    4214 scope.go:122] "RemoveContainer" containerID="5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"
Oct 30 03:28:28.154060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.154043    4214 scope.go:122] "RemoveContainer" containerID="5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"
Oct 30 03:28:28.154406 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:28.154386    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909\": container with ID starting with 5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909 not found: ID does not exist" containerID="5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"
Oct 30 03:28:28.154498 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.154412    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909"} err="failed to get container status \"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909\": rpc error: code = NotFound desc = could not find container \"5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909\": container with ID starting with 5ac052c0638f825bf1cd121929065a2b6e73133c26d1081cec9b16f28bc80909 not found: ID does not exist"
Oct 30 03:28:28.195135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.195113    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:28:28.207343 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:28.207318    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-4b5459-d6cf879f4-hz86q"]
Oct 30 03:28:29.080584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:29.080548    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" path="/var/lib/kubelet/pods/66f45ba8-c10a-4a44-b87d-7f64ef6f1693/volumes"
Oct 30 03:28:29.572596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:29.572558    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:30.073826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.073755    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 03:28:30.160734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.160702    4214 generic.go:356] "Generic (PLEG): container finished" podID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerID="b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f" exitCode=0
Oct 30 03:28:30.161299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.160772    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"
Oct 30 03:28:30.161299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.160791    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" event={"ID":"36c9e5d7-8358-4041-93ea-5d9cc1d1f226","Type":"ContainerDied","Data":"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"}
Oct 30 03:28:30.161299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.160861    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t" event={"ID":"36c9e5d7-8358-4041-93ea-5d9cc1d1f226","Type":"ContainerDied","Data":"6386b88a73a8fe1c739c7a5e0e9dde5de13d98e7fb98eca1b75afb511857a6cc"}
Oct 30 03:28:30.161299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.160894    4214 scope.go:122] "RemoveContainer" containerID="b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"
Oct 30 03:28:30.169345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.169328    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\") pod \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") "
Oct 30 03:28:30.169476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.169458    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\" (UniqueName: \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\") pod \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\" (UID: \"36c9e5d7-8358-4041-93ea-5d9cc1d1f226\") "
Oct 30 03:28:30.169711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.169690    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume" pod "36c9e5d7-8358-4041-93ea-5d9cc1d1f226" (UID: "36c9e5d7-8358-4041-93ea-5d9cc1d1f226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:30.169945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.169923    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.171592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.171571    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx" pod "36c9e5d7-8358-4041-93ea-5d9cc1d1f226" (UID: "36c9e5d7-8358-4041-93ea-5d9cc1d1f226"). InnerVolumeSpecName "kube-api-access-gpqpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:30.173499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.173479    4214 scope.go:122] "RemoveContainer" containerID="b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"
Oct 30 03:28:30.173825 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:30.173797    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f\": container with ID starting with b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f not found: ID does not exist" containerID="b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"
Oct 30 03:28:30.173917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.173834    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f"} err="failed to get container status \"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f\": rpc error: code = NotFound desc = could not find container \"b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f\": container with ID starting with b1dd4a8fc963393f8699818c850f5aedd3979e6647d919f13e7e98d7a944f45f not found: ID does not exist"
Oct 30 03:28:30.271017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.270979    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqpx\" (UniqueName: \"kubernetes.io/projected/36c9e5d7-8358-4041-93ea-5d9cc1d1f226-kube-api-access-gpqpx\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.510950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.510916    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 03:28:30.517090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:30.517066    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e11f03-6c6749f4f5-j7k4t"]
Oct 30 03:28:31.067530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:31.067490    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" path="/var/lib/kubelet/pods/36c9e5d7-8358-4041-93ea-5d9cc1d1f226/volumes"
Oct 30 03:28:33.373348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.373318    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:28:33.509998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.509959    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\" (UniqueName: \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\") pod \"9307f6b3-02e6-43bf-9224-c6256bef88f8\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") "
Oct 30 03:28:33.510282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.510093    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\") pod \"9307f6b3-02e6-43bf-9224-c6256bef88f8\" (UID: \"9307f6b3-02e6-43bf-9224-c6256bef88f8\") "
Oct 30 03:28:33.510492 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.510469    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume" pod "9307f6b3-02e6-43bf-9224-c6256bef88f8" (UID: "9307f6b3-02e6-43bf-9224-c6256bef88f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:33.510644 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.510624    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9307f6b3-02e6-43bf-9224-c6256bef88f8-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.512199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.512178    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9" pod "9307f6b3-02e6-43bf-9224-c6256bef88f8" (UID: "9307f6b3-02e6-43bf-9224-c6256bef88f8"). InnerVolumeSpecName "kube-api-access-2cbf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:33.611628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:33.611563    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbf9\" (UniqueName: \"kubernetes.io/projected/9307f6b3-02e6-43bf-9224-c6256bef88f8-kube-api-access-2cbf9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:34.192328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.192292    4214 generic.go:356] "Generic (PLEG): container finished" podID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerID="146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e" exitCode=0
Oct 30 03:28:34.192510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.192343    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"
Oct 30 03:28:34.192510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.192368    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" event={"ID":"9307f6b3-02e6-43bf-9224-c6256bef88f8","Type":"ContainerDied","Data":"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"}
Oct 30 03:28:34.192510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.192421    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx" event={"ID":"9307f6b3-02e6-43bf-9224-c6256bef88f8","Type":"ContainerDied","Data":"50acf89cc42d04eab4bbc402783c475246c1804953251d94200c9335e3c7403f"}
Oct 30 03:28:34.192510 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.192441    4214 scope.go:122] "RemoveContainer" containerID="146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"
Oct 30 03:28:34.204574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.204558    4214 scope.go:122] "RemoveContainer" containerID="146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"
Oct 30 03:28:34.204864 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:34.204847    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e\": container with ID starting with 146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e not found: ID does not exist" containerID="146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"
Oct 30 03:28:34.204916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.204873    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e"} err="failed to get container status \"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e\": rpc error: code = NotFound desc = could not find container \"146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e\": container with ID starting with 146492f7e4ad6ab368a0b6e77832940902e5d77223b837d9a845e538ba70537e not found: ID does not exist"
Oct 30 03:28:34.249274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.249242    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:28:34.268712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:34.268681    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-4b5459-56c5cd7f4b-rtccx"]
Oct 30 03:28:35.066851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:35.066817    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" path="/var/lib/kubelet/pods/9307f6b3-02e6-43bf-9224-c6256bef88f8/volumes"
Oct 30 03:28:39.165928 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:39.165889    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:28:46.688993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:46.688957    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:28:46.725192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:46.725160    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:28:51.689626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:51.689582    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing" containerID="cri-o://84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0" gracePeriod=25
Oct 30 03:28:51.725725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:51.725684    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing" containerID="cri-o://f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d" gracePeriod=25
Oct 30 03:28:52.433107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:52.433071    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:28:52.433362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:52.433334    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" containerID="cri-o://ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9" gracePeriod=30
Oct 30 03:28:54.408164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.408130    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:28:54.408694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.408393    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" podUID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" containerName="maas-api" containerID="cri-o://229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378" gracePeriod=30
Oct 30 03:28:54.732205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.731704    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:28:54.818749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.818707    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\") pod \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") "
Oct 30 03:28:54.819060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.819034    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\") pod \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") "
Oct 30 03:28:54.819297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.819279    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\") pod \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") "
Oct 30 03:28:54.819544 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.819526    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\" (UniqueName: \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\") pod \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\" (UID: \"3b2f09ee-73ed-49a9-8ec6-6e0f96982882\") "
Oct 30 03:28:54.824330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.824084    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls" pod "3b2f09ee-73ed-49a9-8ec6-6e0f96982882" (UID: "3b2f09ee-73ed-49a9-8ec6-6e0f96982882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:54.824811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.824562    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca" pod "3b2f09ee-73ed-49a9-8ec6-6e0f96982882" (UID: "3b2f09ee-73ed-49a9-8ec6-6e0f96982882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:54.867426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.867360    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs" pod "3b2f09ee-73ed-49a9-8ec6-6e0f96982882" (UID: "3b2f09ee-73ed-49a9-8ec6-6e0f96982882"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:54.875443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.875405    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb" pod "3b2f09ee-73ed-49a9-8ec6-6e0f96982882" (UID: "3b2f09ee-73ed-49a9-8ec6-6e0f96982882"). InnerVolumeSpecName "kube-api-access-x9bjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:54.922034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.921998    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:54.922034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.922031    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bjb\" (UniqueName: \"kubernetes.io/projected/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-kube-api-access-x9bjb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:54.922034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.922042    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:54.922309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:54.922050    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f09ee-73ed-49a9-8ec6-6e0f96982882-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:55.378597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.378502    4214 generic.go:356] "Generic (PLEG): container finished" podID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" containerID="229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378" exitCode=0
Oct 30 03:28:55.378597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.378554    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"
Oct 30 03:28:55.378854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.378610    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" event={"ID":"3b2f09ee-73ed-49a9-8ec6-6e0f96982882","Type":"ContainerDied","Data":"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"}
Oct 30 03:28:55.378854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.378681    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6" event={"ID":"3b2f09ee-73ed-49a9-8ec6-6e0f96982882","Type":"ContainerDied","Data":"0ef665a948faaa7e2c92b3903dec12a9e8f1dcfc2da8fa41c139192f3f2afc54"}
Oct 30 03:28:55.378854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.378730    4214 scope.go:122] "RemoveContainer" containerID="229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"
Oct 30 03:28:55.389403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.389384    4214 scope.go:122] "RemoveContainer" containerID="229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"
Oct 30 03:28:55.389684 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:55.389663    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378\": container with ID starting with 229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378 not found: ID does not exist" containerID="229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"
Oct 30 03:28:55.389729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.389691    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378"} err="failed to get container status \"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378\": rpc error: code = NotFound desc = could not find container \"229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378\": container with ID starting with 229fa9adeb83ac04bc489acec07d6946c5753a6db6ff5a7a7f3320802cd6b378 not found: ID does not exist"
Oct 30 03:28:55.433993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.433971    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:28:55.437303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.437279    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-8b0787-744cbf5b48-kmqz6"]
Oct 30 03:28:55.668581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.668418    4214 patch_prober.go:29] interesting pod/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:28:55.668581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.668476    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:28:55.959717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.959627    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 03:28:55.959933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:55.959862    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy" containerID="cri-o://cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc" gracePeriod=30
Oct 30 03:28:57.085201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.085158    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" path="/var/lib/kubelet/pods/3b2f09ee-73ed-49a9-8ec6-6e0f96982882/volumes"
Oct 30 03:28:57.739254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.739217    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:28:57.858873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.858830    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859076    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859165    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859206    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859265    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859295    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859326    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859362    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859412    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:57.859588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859469    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\") pod \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\" (UID: \"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0\") "
Oct 30 03:28:57.859794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859704    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:57.859852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859787    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:57.859931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.859903    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:57.860038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.860016    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:57.860166 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.860148    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.860306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.860186    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.860306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.860201    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.860306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.860216    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.862158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.862131    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:57.862318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.862132    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:57.862318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.862224    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:57.862415 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.862325    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj" pod "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" (UID: "cfd1b954-857a-4b97-8e07-2fdbf90d9cd0"). InnerVolumeSpecName "kube-api-access-9lwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:57.961630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.961556    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.961630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.961585    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.961630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.961595    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.961630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.961605    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwsj\" (UniqueName: \"kubernetes.io/projected/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-kube-api-access-9lwsj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:57.961630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:57.961615    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:28:58.404572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.404534    4214 generic.go:356] "Generic (PLEG): container finished" podID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerID="ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9" exitCode=0
Oct 30 03:28:58.405051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.404592    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"
Oct 30 03:28:58.405051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.404623    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerDied","Data":"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"}
Oct 30 03:28:58.405051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.404700    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6" event={"ID":"cfd1b954-857a-4b97-8e07-2fdbf90d9cd0","Type":"ContainerDied","Data":"a1b031b0fa61329f8dafa87ef0fa2fdfdeac97fce401f6ef11a19eb45847b74a"}
Oct 30 03:28:58.405051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.404734    4214 scope.go:122] "RemoveContainer" containerID="ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"
Oct 30 03:28:58.415848 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.415829    4214 scope.go:122] "RemoveContainer" containerID="50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"
Oct 30 03:28:58.425079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.425063    4214 scope.go:122] "RemoveContainer" containerID="ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"
Oct 30 03:28:58.425407 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:58.425386    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9\": container with ID starting with ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9 not found: ID does not exist" containerID="ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"
Oct 30 03:28:58.425476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.425416    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9"} err="failed to get container status \"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9\": rpc error: code = NotFound desc = could not find container \"ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9\": container with ID starting with ccaf3216e4b845da315512a1697dd2a94b3e5915af0b9d9ae8a94e866400a5d9 not found: ID does not exist"
Oct 30 03:28:58.425476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.425433    4214 scope.go:122] "RemoveContainer" containerID="50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"
Oct 30 03:28:58.425660 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:28:58.425637    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f\": container with ID starting with 50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f not found: ID does not exist" containerID="50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"
Oct 30 03:28:58.425719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.425668    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f"} err="failed to get container status \"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f\": rpc error: code = NotFound desc = could not find container \"50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f\": container with ID starting with 50a36f61f5734f8b042905e9aff8e2bbe253292d44ced04c952647f2b605c53f not found: ID does not exist"
Oct 30 03:28:58.474052 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.474027    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:28:58.479749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:58.479723    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9528f-openshift-default-5c7c76bd9c-pwjm6"]
Oct 30 03:28:59.066887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:59.066857    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" path="/var/lib/kubelet/pods/cfd1b954-857a-4b97-8e07-2fdbf90d9cd0/volumes"
Oct 30 03:28:59.370340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:59.370252    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:59.370493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:28:59.370262    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:01.242835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.242774    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 03:29:01.412137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412107    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412182    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412213    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412249    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412297    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412348    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412446    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412487    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412503    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412539    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.412579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412574    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.412919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412627    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\") pod \"e9cf2197-3df5-43f9-b858-23e03e0db025\" (UID: \"e9cf2197-3df5-43f9-b858-23e03e0db025\") "
Oct 30 03:29:01.412919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412767    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.412919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.412853    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:01.413170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.413146    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.413267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.413175    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.413267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.413191    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.413267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.413204    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.413267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.413216    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cf2197-3df5-43f9-b858-23e03e0db025-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.414767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.414738    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "kube-api-access-52wq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:01.414871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.414771    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.414871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.414799    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:01.414984 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.414925    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo" pod "e9cf2197-3df5-43f9-b858-23e03e0db025" (UID: "e9cf2197-3df5-43f9-b858-23e03e0db025"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:01.428190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.428161    4214 generic.go:356] "Generic (PLEG): container finished" podID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerID="cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc" exitCode=0
Oct 30 03:29:01.428326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.428214    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"
Oct 30 03:29:01.428326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.428254    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" event={"ID":"e9cf2197-3df5-43f9-b858-23e03e0db025","Type":"ContainerDied","Data":"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"}
Oct 30 03:29:01.428454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.428345    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m" event={"ID":"e9cf2197-3df5-43f9-b858-23e03e0db025","Type":"ContainerDied","Data":"69c77f0c0fa261e4102f2c0941669d7039f841658d01d53f2e6b4f9f89c1dfed"}
Oct 30 03:29:01.428454 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.428380    4214 scope.go:122] "RemoveContainer" containerID="cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"
Oct 30 03:29:01.442629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.442613    4214 scope.go:122] "RemoveContainer" containerID="cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"
Oct 30 03:29:01.442896 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:01.442878    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc\": container with ID starting with cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc not found: ID does not exist" containerID="cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"
Oct 30 03:29:01.442951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.442903    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc"} err="failed to get container status \"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc\": rpc error: code = NotFound desc = could not find container \"cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc\": container with ID starting with cb409f94784a621753d58b5d5e0cfeee690beea8d7979fb82b91284d8ef709dc not found: ID does not exist"
Oct 30 03:29:01.483430 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.483401    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 03:29:01.487716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.487691    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-r2t9m"]
Oct 30 03:29:01.515092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.515025    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wq8\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-kube-api-access-52wq8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.515092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.515057    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.515092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.515071    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.515092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:01.515085    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cf2197-3df5-43f9-b858-23e03e0db025-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:02.143621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:02.143590    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:29:02.144039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:02.144007    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" containerID="cri-o://1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41" gracePeriod=30
Oct 30 03:29:03.070048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:03.070013    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" path="/var/lib/kubelet/pods/e9cf2197-3df5-43f9-b858-23e03e0db025/volumes"
Oct 30 03:29:03.955269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:03.955206    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:29:03.955474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:03.955291    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:29:07.424635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.424609    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:29:07.480168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.479468    4214 generic.go:356] "Generic (PLEG): container finished" podID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerID="1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41" exitCode=0
Oct 30 03:29:07.480681 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.479542    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"
Oct 30 03:29:07.482549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.479561    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerDied","Data":"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"}
Oct 30 03:29:07.482709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.482584    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9" event={"ID":"7454b33a-0272-4cad-a8f2-8d2b1a939917","Type":"ContainerDied","Data":"ace11b46d07b8168d9ed03cda66170f2e67692423e5a7ef1288063f98aa844a2"}
Oct 30 03:29:07.482709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.482626    4214 scope.go:122] "RemoveContainer" containerID="1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"
Oct 30 03:29:07.493596 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.493567    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.493835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.493713    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.493953    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494015    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494066    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494112    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494182    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494210    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494193    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:07.494461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494267    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494479    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:07.494893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494714    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:07.494893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494753    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\") pod \"7454b33a-0272-4cad-a8f2-8d2b1a939917\" (UID: \"7454b33a-0272-4cad-a8f2-8d2b1a939917\") "
Oct 30 03:29:07.494893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.494777    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:07.495259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.495222    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.495311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.495267    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.495311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.495282    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.495311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.495295    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7454b33a-0272-4cad-a8f2-8d2b1a939917-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.495311 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.495306    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.496815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.496719    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "kube-api-access-skks4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:07.496882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.496866    4214 scope.go:122] "RemoveContainer" containerID="06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"
Oct 30 03:29:07.497119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.497098    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:07.498295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.498269    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:07.499162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.499139    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo" pod "7454b33a-0272-4cad-a8f2-8d2b1a939917" (UID: "7454b33a-0272-4cad-a8f2-8d2b1a939917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:07.521815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.521790    4214 scope.go:122] "RemoveContainer" containerID="1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"
Oct 30 03:29:07.522084 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:07.522063    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41\": container with ID starting with 1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41 not found: ID does not exist" containerID="1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"
Oct 30 03:29:07.522179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.522093    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41"} err="failed to get container status \"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41\": rpc error: code = NotFound desc = could not find container \"1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41\": container with ID starting with 1aad3ada2a16f327c0a37da0fac42062ccefe5f17539e48db6fcf83ed4649a41 not found: ID does not exist"
Oct 30 03:29:07.522179 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.522111    4214 scope.go:122] "RemoveContainer" containerID="06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"
Oct 30 03:29:07.522379 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:07.522362    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7\": container with ID starting with 06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7 not found: ID does not exist" containerID="06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"
Oct 30 03:29:07.522499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.522384    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7"} err="failed to get container status \"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7\": rpc error: code = NotFound desc = could not find container \"06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7\": container with ID starting with 06b42a844819124a6b2880bcad869d450d008f162e48ea4c1142a6016a1896b7 not found: ID does not exist"
Oct 30 03:29:07.591003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.590955    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 03:29:07.596572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.596546    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.596572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.596571    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.596750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.596581    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skks4\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-kube-api-access-skks4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.596750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.596591    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7454b33a-0272-4cad-a8f2-8d2b1a939917-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:07.831101 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.831071    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:29:07.835385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:07.835356    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-8b0787-openshift-default-66b56f74b5-4vpg9"]
Oct 30 03:29:09.068667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:09.068623    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" path="/var/lib/kubelet/pods/7454b33a-0272-4cad-a8f2-8d2b1a939917/volumes"
Oct 30 03:29:09.170221 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:09.170183    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:29:09.369455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:09.369367    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:09.369840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:09.369808    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:12.022922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.022893    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:29:12.026570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.026543    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:29:12.052972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.052947    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\" (UniqueName: \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\") pod \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") "
Oct 30 03:29:12.053169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053156    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\") pod \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") "
Oct 30 03:29:12.053258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053243    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\" (UniqueName: \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\") pod \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\" (UID: \"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4\") "
Oct 30 03:29:12.053351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053335    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\") pod \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\" (UID: \"1625ec04-2af7-4911-8dfd-5cb34f4debc8\") "
Oct 30 03:29:12.053736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053710    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume" pod "6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" (UID: "6b63c816-f9fe-4cb8-8d7f-1d1089f940d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:12.053845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053815    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:12.053845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.053820    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume" pod "1625ec04-2af7-4911-8dfd-5cb34f4debc8" (UID: "1625ec04-2af7-4911-8dfd-5cb34f4debc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:12.055580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.055562    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn" pod "1625ec04-2af7-4911-8dfd-5cb34f4debc8" (UID: "1625ec04-2af7-4911-8dfd-5cb34f4debc8"). InnerVolumeSpecName "kube-api-access-z5hgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:12.055923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.055904    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t" pod "6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" (UID: "6b63c816-f9fe-4cb8-8d7f-1d1089f940d4"). InnerVolumeSpecName "kube-api-access-r7s7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:12.155098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.155056    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s7t\" (UniqueName: \"kubernetes.io/projected/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4-kube-api-access-r7s7t\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:12.155098 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.155098    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1625ec04-2af7-4911-8dfd-5cb34f4debc8-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:12.155347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.155113    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5hgn\" (UniqueName: \"kubernetes.io/projected/1625ec04-2af7-4911-8dfd-5cb34f4debc8-kube-api-access-z5hgn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:12.516190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.516153    4214 generic.go:356] "Generic (PLEG): container finished" podID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerID="f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d" exitCode=0
Oct 30 03:29:12.516401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.516202    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"
Oct 30 03:29:12.516401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.516258    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" event={"ID":"1625ec04-2af7-4911-8dfd-5cb34f4debc8","Type":"ContainerDied","Data":"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"}
Oct 30 03:29:12.516401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.516311    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh" event={"ID":"1625ec04-2af7-4911-8dfd-5cb34f4debc8","Type":"ContainerDied","Data":"3694b80061f7b0aa7d6121a31cfb8ccef727c6ae3f228106f8e8f110b7469dda"}
Oct 30 03:29:12.516401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.516333    4214 scope.go:122] "RemoveContainer" containerID="f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"
Oct 30 03:29:12.517760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.517734    4214 generic.go:356] "Generic (PLEG): container finished" podID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerID="84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0" exitCode=0
Oct 30 03:29:12.517881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.517764    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" event={"ID":"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4","Type":"ContainerDied","Data":"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"}
Oct 30 03:29:12.517881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.517782    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"
Oct 30 03:29:12.517881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.517816    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7" event={"ID":"6b63c816-f9fe-4cb8-8d7f-1d1089f940d4","Type":"ContainerDied","Data":"23ee05b68f0aa3f987fcdc5d4f0d093494f0b50a121a301888d031ff98feebf2"}
Oct 30 03:29:12.528517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.528485    4214 scope.go:122] "RemoveContainer" containerID="f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"
Oct 30 03:29:12.528787 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:12.528763    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d\": container with ID starting with f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d not found: ID does not exist" containerID="f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"
Oct 30 03:29:12.528885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.528794    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d"} err="failed to get container status \"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d\": rpc error: code = NotFound desc = could not find container \"f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d\": container with ID starting with f91eeb38eb30ff3f9077d6fb3088cf8588447b9e48382a3a0d50fcb72a48e49d not found: ID does not exist"
Oct 30 03:29:12.528885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.528815    4214 scope.go:122] "RemoveContainer" containerID="84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"
Oct 30 03:29:12.537606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.537591    4214 scope.go:122] "RemoveContainer" containerID="84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"
Oct 30 03:29:12.537832 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:12.537814    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0\": container with ID starting with 84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0 not found: ID does not exist" containerID="84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"
Oct 30 03:29:12.537903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.537853    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0"} err="failed to get container status \"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0\": rpc error: code = NotFound desc = could not find container \"84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0\": container with ID starting with 84fea2b6a0b035229d810da24da790bcbacc34d92dc0bd96f047c2adeb0057b0 not found: ID does not exist"
Oct 30 03:29:12.570568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.570533    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:29:12.582749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.582724    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-8b0787-769b97b9fd-l6gx7"]
Oct 30 03:29:12.617792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.617764    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:29:12.620841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:12.620817    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-8b0787-7f6dc7b6c5-pbpfh"]
Oct 30 03:29:13.068756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:13.068712    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" path="/var/lib/kubelet/pods/1625ec04-2af7-4911-8dfd-5cb34f4debc8/volumes"
Oct 30 03:29:13.069318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:13.069290    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" path="/var/lib/kubelet/pods/6b63c816-f9fe-4cb8-8d7f-1d1089f940d4/volumes"
Oct 30 03:29:13.266250 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:13.266193    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 03:29:13.299331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:13.299293    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 03:29:14.297487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.297450    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:29:14.297996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.297687    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerName="maas-api" containerID="cri-o://4ba3e23b5db557735c2162b95d5114e8d526f4e2e2bc906f0f8479bd4ecb49aa" gracePeriod=30
Oct 30 03:29:14.431792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.431754    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.38:8443/health\": dial tcp 10.133.1.38:8443: connect: connection refused"
Oct 30 03:29:14.588300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.585277    4214 generic.go:356] "Generic (PLEG): container finished" podID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerID="4ba3e23b5db557735c2162b95d5114e8d526f4e2e2bc906f0f8479bd4ecb49aa" exitCode=0
Oct 30 03:29:14.588300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.585503    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" event={"ID":"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1","Type":"ContainerDied","Data":"4ba3e23b5db557735c2162b95d5114e8d526f4e2e2bc906f0f8479bd4ecb49aa"}
Oct 30 03:29:14.630029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.630004    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:29:14.690635 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.690597    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\") pod \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") "
Oct 30 03:29:14.690844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.690670    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\" (UniqueName: \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\") pod \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") "
Oct 30 03:29:14.690844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.690732    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\") pod \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") "
Oct 30 03:29:14.690997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.690901    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\") pod \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\" (UID: \"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1\") "
Oct 30 03:29:14.691357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.691061    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca" pod "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" (UID: "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:14.692264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.691571    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.693756 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.693732    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls" pod "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" (UID: "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:14.693888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.693798    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs" pod "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" (UID: "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:14.693965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.693890    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264" pod "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" (UID: "3d93fa35-c0e3-49f4-8f7d-adc83dc710a1"). InnerVolumeSpecName "kube-api-access-5z264". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:14.792338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.792297    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z264\" (UniqueName: \"kubernetes.io/projected/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-kube-api-access-5z264\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.792338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.792334    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.792338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:14.792346    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:15.592968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:15.592937    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"
Oct 30 03:29:15.592968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:15.592955    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw" event={"ID":"3d93fa35-c0e3-49f4-8f7d-adc83dc710a1","Type":"ContainerDied","Data":"3e4c1ac1bb2ef154237eebce08a1fe36325416c0a76d9bd20e479295c597f592"}
Oct 30 03:29:15.593523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:15.593021    4214 scope.go:122] "RemoveContainer" containerID="4ba3e23b5db557735c2162b95d5114e8d526f4e2e2bc906f0f8479bd4ecb49aa"
Oct 30 03:29:15.648615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:15.648583    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:29:15.661546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:15.661512    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-055daa-84b9854f4-ds7fw"]
Oct 30 03:29:17.068144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:17.068108    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" path="/var/lib/kubelet/pods/3d93fa35-c0e3-49f4-8f7d-adc83dc710a1/volumes"
Oct 30 03:29:18.266578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:18.266535    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing" containerID="cri-o://6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059" gracePeriod=25
Oct 30 03:29:18.299728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:18.299677    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing" containerID="cri-o://ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1" gracePeriod=25
Oct 30 03:29:21.466822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.466778    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:21.466822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.466801    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:21.658629 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.658598    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:29:21.658976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.658933    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" containerID="cri-o://7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f" gracePeriod=30
Oct 30 03:29:21.717712 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.717633    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:29:21.718340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718321    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing"
Oct 30 03:29:21.718340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718339    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718356    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718364    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718379    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718386    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718396    4214 cpu_manager.go:414] "containerMap: removing container" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718403    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718420    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718427    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718437    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718445    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718455    4214 cpu_manager.go:414] "containerMap: removing container" podUID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" containerName="maas-api"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718462    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" containerName="maas-api"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718474    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718481    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718493    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718501    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718523    4214 cpu_manager.go:414] "containerMap: removing container" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718531    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing"
Oct 30 03:29:21.718539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718541    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718548    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718562    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e8777733-f636-4161-915c-b01fd20df369" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718569    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8777733-f636-4161-915c-b01fd20df369" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718580    4214 cpu_manager.go:414] "containerMap: removing container" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718589    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718608    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718615    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718639    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718645    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718659    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718666    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718678    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718684    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718693    4214 cpu_manager.go:414] "containerMap: removing container" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718700    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718709    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718716    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718730    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2618539f-c69f-43ac-8db3-30704cfaf7cb" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718737    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2618539f-c69f-43ac-8db3-30704cfaf7cb" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718879    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc5f5b9-e941-4236-aae5-3c4059863209" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718895    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ce25d3-ead3-45dc-b5d4-fa157d141a01" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718905    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d93fa35-c0e3-49f4-8f7d-adc83dc710a1" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718915    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b63c816-f9fe-4cb8-8d7f-1d1089f940d4" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718924    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde07f9d-958a-4478-9c16-08d908902b66" containerName="main"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718931    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2f09ee-73ed-49a9-8ec6-6e0f96982882" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718941    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2618539f-c69f-43ac-8db3-30704cfaf7cb" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718950    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718958    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18381b7-5795-4c53-954c-33859dee342b" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718970    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9307f6b3-02e6-43bf-9224-c6256bef88f8" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718981    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c4af4c-7c45-440f-88b8-066a5020071d" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.718999    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719009    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd1b954-857a-4b97-8e07-2fdbf90d9cd0" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719018    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f45ba8-c10a-4a44-b87d-7f64ef6f1693" containerName="maas-api"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719029    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf2197-3df5-43f9-b858-23e03e0db025" containerName="istio-proxy"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719039    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c9e5d7-8358-4041-93ea-5d9cc1d1f226" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719049    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1625ec04-2af7-4911-8dfd-5cb34f4debc8" containerName="payload-pre-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719058    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8777733-f636-4161-915c-b01fd20df369" containerName="payload-processing"
Oct 30 03:29:21.719637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.719067    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="630f8512-3070-42d8-8bb3-fea98031b56c" containerName="payload-pre-processing"
Oct 30 03:29:21.722867 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.722847    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.738406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.738382    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cf605a\"/\"default-dockercfg-mt5k2\""
Oct 30 03:29:21.738548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.738411    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cf605a\"/\"kube-root-ca.crt\""
Oct 30 03:29:21.738548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.738411    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cf605a\"/\"openshift-service-ca.crt\""
Oct 30 03:29:21.738651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.738563    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cf605a\"/\"ipp-route-cf605a-kserve-self-signed-certs\""
Oct 30 03:29:21.739168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.739152    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:29:21.805740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.805706    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.805907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.805748    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.805907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.805881    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.806027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.805939    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.806105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.806082    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.806165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.806129    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzb2\" (UniqueName: \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907477    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907565    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907596    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzb2\" (UniqueName: \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907667    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907691    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.907956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.907786    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.908213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.908185    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.908386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.908213    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.908386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.908331    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.909839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.909821    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.910139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.910121    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:21.930087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:21.928727    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzb2\" (UniqueName: \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\") pod \"ipp-route-cf605a-kserve-78779c7665-sj452\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") " pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:22.035597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.035526    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:22.342771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.342741    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:29:22.345275 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:29:22.345246    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a3e0c57_eed0_4005_99f5_3c03171418d2.slice/crio-91bdc3037d11c658d36d9f140bc6587913e823b6d4b7f8af42f81b3fa3890820 WatchSource:0}: Error finding container 91bdc3037d11c658d36d9f140bc6587913e823b6d4b7f8af42f81b3fa3890820: Status 404 returned error can't find the container with id 91bdc3037d11c658d36d9f140bc6587913e823b6d4b7f8af42f81b3fa3890820
Oct 30 03:29:22.612993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.612905    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" containerID="cri-o://b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8" gracePeriod=45
Oct 30 03:29:22.651429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.651368    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" event={"ID":"6a3e0c57-eed0-4005-99f5-3c03171418d2","Type":"ContainerStarted","Data":"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"}
Oct 30 03:29:22.651429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.651435    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" event={"ID":"6a3e0c57-eed0-4005-99f5-3c03171418d2","Type":"ContainerStarted","Data":"91bdc3037d11c658d36d9f140bc6587913e823b6d4b7f8af42f81b3fa3890820"}
Oct 30 03:29:22.693254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.693187    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" podStartSLOduration=1.693170729 podStartE2EDuration="1.693170729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:22.691485283 +0000 UTC m=+2901.765656068" watchObservedRunningTime="2026-10-30 03:29:22.693170729 +0000 UTC m=+2901.767341522"
Oct 30 03:29:22.699080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.699049    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Oct 30 03:29:22.891325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:22.891296    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:29:23.031426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031388    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.031613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031445    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\" (UniqueName: \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.031613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031519    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.031758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031610    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.031758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031719    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.031876 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031783    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.031940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031866    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.031940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.031888    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\") pod \"93a18368-7423-47c1-b0dc-c2aaf3999a42\" (UID: \"93a18368-7423-47c1-b0dc-c2aaf3999a42\") "
Oct 30 03:29:23.032437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.032078    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.032437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.032372    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.032437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.032394    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.032437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.032407    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.034203 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.034181    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.034475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.034450    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "kube-api-access-nn4kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:23.035310 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.035281    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm" pod "93a18368-7423-47c1-b0dc-c2aaf3999a42" (UID: "93a18368-7423-47c1-b0dc-c2aaf3999a42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.036317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.036294    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:23.133723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.133653    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a18368-7423-47c1-b0dc-c2aaf3999a42-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.133723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.133681    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4kv\" (UniqueName: \"kubernetes.io/projected/93a18368-7423-47c1-b0dc-c2aaf3999a42-kube-api-access-nn4kv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.133723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.133690    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a18368-7423-47c1-b0dc-c2aaf3999a42-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.661802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.661760    4214 generic.go:356] "Generic (PLEG): container finished" podID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerID="b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8" exitCode=0
Oct 30 03:29:23.662329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.661811    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"
Oct 30 03:29:23.662329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.661842    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" event={"ID":"93a18368-7423-47c1-b0dc-c2aaf3999a42","Type":"ContainerDied","Data":"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"}
Oct 30 03:29:23.662329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.661886    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t" event={"ID":"93a18368-7423-47c1-b0dc-c2aaf3999a42","Type":"ContainerDied","Data":"18c249b6beba35e8d303c2c545751ba5be02d6808c1fa245fea1185e2d8ff8af"}
Oct 30 03:29:23.662329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.661903    4214 scope.go:122] "RemoveContainer" containerID="b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"
Oct 30 03:29:23.672309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.672282    4214 scope.go:122] "RemoveContainer" containerID="b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"
Oct 30 03:29:23.672589 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:23.672571    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8\": container with ID starting with b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8 not found: ID does not exist" containerID="b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"
Oct 30 03:29:23.672646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.672597    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8"} err="failed to get container status \"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8\": rpc error: code = NotFound desc = could not find container \"b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8\": container with ID starting with b6aeb918cae25e93d26fb93b6844b42a1224c3dcf78c50a516a3634f8e3221a8 not found: ID does not exist"
Oct 30 03:29:23.717084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.717054    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 03:29:23.729675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.729648    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/premium-w6-32bad3-kserve-5c8c79f499-n6d6t"]
Oct 30 03:29:23.963554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.963470    4214 patch_prober.go:29] interesting pod/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:29:23.963554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:23.963518    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:29:25.067127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:25.067095    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" path="/var/lib/kubelet/pods/93a18368-7423-47c1-b0dc-c2aaf3999a42/volumes"
Oct 30 03:29:26.916270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.916214    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:29:26.979893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.979800    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.979908    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.979955    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.979983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980074    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980144    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980176    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980210    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980285    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\") pod \"015c99fd-736c-4585-a94e-c20ecac4f1f4\" (UID: \"015c99fd-736c-4585-a94e-c20ecac4f1f4\") "
Oct 30 03:29:26.980572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980510    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:26.980739 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.980722    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:26.982073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.982042    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:26.982225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.982204    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:26.982654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.982590    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:26.982792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.982771    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:26.984349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.984313    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "kube-api-access-zxmf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:26.985286 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.985264    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:26.985633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.985603    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:26.986482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:26.986446    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token" pod "015c99fd-736c-4585-a94e-c20ecac4f1f4" (UID: "015c99fd-736c-4585-a94e-c20ecac4f1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:27.082384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082346    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082374    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082391    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015c99fd-736c-4585-a94e-c20ecac4f1f4-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082405    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082417    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmf6\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-kube-api-access-zxmf6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082429    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082442    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.082568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.082453    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015c99fd-736c-4585-a94e-c20ecac4f1f4-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:27.693483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.693444    4214 generic.go:356] "Generic (PLEG): container finished" podID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerID="7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f" exitCode=0
Oct 30 03:29:27.693685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.693511    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"
Oct 30 03:29:27.693685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.693525    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerDied","Data":"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"}
Oct 30 03:29:27.693685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.693587    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2" event={"ID":"015c99fd-736c-4585-a94e-c20ecac4f1f4","Type":"ContainerDied","Data":"0626aa2c995da85fa2f8c08ff28b556aec622317785539a87f7c2429c175ddc9"}
Oct 30 03:29:27.693685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.693631    4214 scope.go:122] "RemoveContainer" containerID="7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"
Oct 30 03:29:27.703704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.703685    4214 scope.go:122] "RemoveContainer" containerID="26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"
Oct 30 03:29:27.713031 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.713008    4214 scope.go:122] "RemoveContainer" containerID="7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"
Oct 30 03:29:27.713369 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:27.713348    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f\": container with ID starting with 7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f not found: ID does not exist" containerID="7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"
Oct 30 03:29:27.713476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.713383    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f"} err="failed to get container status \"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f\": rpc error: code = NotFound desc = could not find container \"7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f\": container with ID starting with 7ba8c6200384913c7ee7215d92cfa66242eb41211e641baab6ac3e9f82cd896f not found: ID does not exist"
Oct 30 03:29:27.713476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.713411    4214 scope.go:122] "RemoveContainer" containerID="26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"
Oct 30 03:29:27.713726 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:27.713694    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca\": container with ID starting with 26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca not found: ID does not exist" containerID="26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"
Oct 30 03:29:27.713847 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.713733    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca"} err="failed to get container status \"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca\": rpc error: code = NotFound desc = could not find container \"26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca\": container with ID starting with 26a4333451d07368c6cc84767220ba4e2dea50cf3299efaa35b78af8dfde9aca not found: ID does not exist"
Oct 30 03:29:27.763870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.763832    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:29:27.774624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:27.774592    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-055daa-openshift-default-68c775fb45-pdrd2"]
Oct 30 03:29:29.066971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:29.066925    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" path="/var/lib/kubelet/pods/015c99fd-736c-4585-a94e-c20ecac4f1f4/volumes"
Oct 30 03:29:31.466173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:31.466129    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:31.466173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:31.466158    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:32.036171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:32.036137    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:32.049114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:32.049084    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:32.618594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:32.618551    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:29:32.650378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:32.650340    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:29:32.742618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:32.742576    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:29:37.619874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:37.619826    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing" containerID="cri-o://77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31" gracePeriod=25
Oct 30 03:29:37.651375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:37.651325    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing" containerID="cri-o://418db4de87babcdf886cac8cc71248faf2bdbf3ed9def61127b5a26754ba3ba3" gracePeriod=25
Oct 30 03:29:37.734960 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:37.734921    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:29:37.735202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:37.735175    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" podUID="328703a5-2224-4370-a53c-fb9b59915ab2" containerName="maas-api" containerID="cri-o://f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d" gracePeriod=30
Oct 30 03:29:38.037243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.037206    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:29:38.116679 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.116609    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\") pod \"328703a5-2224-4370-a53c-fb9b59915ab2\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") "
Oct 30 03:29:38.116886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.116823    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\") pod \"328703a5-2224-4370-a53c-fb9b59915ab2\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") "
Oct 30 03:29:38.116886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.116856    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\" (UniqueName: \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\") pod \"328703a5-2224-4370-a53c-fb9b59915ab2\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") "
Oct 30 03:29:38.117012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.116894    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\") pod \"328703a5-2224-4370-a53c-fb9b59915ab2\" (UID: \"328703a5-2224-4370-a53c-fb9b59915ab2\") "
Oct 30 03:29:38.125010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.124665    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca" pod "328703a5-2224-4370-a53c-fb9b59915ab2" (UID: "328703a5-2224-4370-a53c-fb9b59915ab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:38.127926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.127456    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls" pod "328703a5-2224-4370-a53c-fb9b59915ab2" (UID: "328703a5-2224-4370-a53c-fb9b59915ab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:38.138849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.137806    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs" pod "328703a5-2224-4370-a53c-fb9b59915ab2" (UID: "328703a5-2224-4370-a53c-fb9b59915ab2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:38.158874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.158788    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn" pod "328703a5-2224-4370-a53c-fb9b59915ab2" (UID: "328703a5-2224-4370-a53c-fb9b59915ab2"). InnerVolumeSpecName "kube-api-access-nxxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:38.218873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.218840    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.218873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.218868    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/328703a5-2224-4370-a53c-fb9b59915ab2-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.218873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.218877    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxmn\" (UniqueName: \"kubernetes.io/projected/328703a5-2224-4370-a53c-fb9b59915ab2-kube-api-access-nxxmn\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.219142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.218888    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/328703a5-2224-4370-a53c-fb9b59915ab2-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.526136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.525765    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 03:29:38.599007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.598965    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 03:29:38.624014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.623983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\") pod \"7ba075b4-a86a-4c49-adfd-5063f81a084e\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") "
Oct 30 03:29:38.624452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.624077    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\" (UniqueName: \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\") pod \"7ba075b4-a86a-4c49-adfd-5063f81a084e\" (UID: \"7ba075b4-a86a-4c49-adfd-5063f81a084e\") "
Oct 30 03:29:38.624452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.624414    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume" pod "7ba075b4-a86a-4c49-adfd-5063f81a084e" (UID: "7ba075b4-a86a-4c49-adfd-5063f81a084e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:38.626136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.626106    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq" pod "7ba075b4-a86a-4c49-adfd-5063f81a084e" (UID: "7ba075b4-a86a-4c49-adfd-5063f81a084e"). InnerVolumeSpecName "kube-api-access-fxgzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:38.725550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.725456    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\" (UniqueName: \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\") pod \"b830bf8a-554a-4597-ad34-9785136e7044\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") "
Oct 30 03:29:38.725550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.725512    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\") pod \"b830bf8a-554a-4597-ad34-9785136e7044\" (UID: \"b830bf8a-554a-4597-ad34-9785136e7044\") "
Oct 30 03:29:38.725896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.725876    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxgzq\" (UniqueName: \"kubernetes.io/projected/7ba075b4-a86a-4c49-adfd-5063f81a084e-kube-api-access-fxgzq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.725957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.725900    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba075b4-a86a-4c49-adfd-5063f81a084e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.725957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.725920    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume" pod "b830bf8a-554a-4597-ad34-9785136e7044" (UID: "b830bf8a-554a-4597-ad34-9785136e7044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:38.727851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.727820    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4" pod "b830bf8a-554a-4597-ad34-9785136e7044" (UID: "b830bf8a-554a-4597-ad34-9785136e7044"). InnerVolumeSpecName "kube-api-access-7xlz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:38.778583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.778549    4214 generic.go:356] "Generic (PLEG): container finished" podID="b830bf8a-554a-4597-ad34-9785136e7044" containerID="6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059" exitCode=0
Oct 30 03:29:38.778763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.778599    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"
Oct 30 03:29:38.778763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.778642    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" event={"ID":"b830bf8a-554a-4597-ad34-9785136e7044","Type":"ContainerDied","Data":"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"}
Oct 30 03:29:38.778763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.778696    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl" event={"ID":"b830bf8a-554a-4597-ad34-9785136e7044","Type":"ContainerDied","Data":"63a9b9462affcb0669de9bbdfb8347d8cb3374fe1294cc5a5cd749bf79136f93"}
Oct 30 03:29:38.778763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.778726    4214 scope.go:122] "RemoveContainer" containerID="6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"
Oct 30 03:29:38.780684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.780662    4214 generic.go:356] "Generic (PLEG): container finished" podID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerID="ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1" exitCode=0
Oct 30 03:29:38.780779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.780706    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"
Oct 30 03:29:38.780779 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.780742    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" event={"ID":"7ba075b4-a86a-4c49-adfd-5063f81a084e","Type":"ContainerDied","Data":"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"}
Oct 30 03:29:38.780904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.780796    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m" event={"ID":"7ba075b4-a86a-4c49-adfd-5063f81a084e","Type":"ContainerDied","Data":"0b7536e8c969ee7ab84aa3972d452df5465d14c2bb1a7fa27f6c08a991ef8887"}
Oct 30 03:29:38.782241 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.782203    4214 generic.go:356] "Generic (PLEG): container finished" podID="328703a5-2224-4370-a53c-fb9b59915ab2" containerID="f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d" exitCode=0
Oct 30 03:29:38.782352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.782264    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"
Oct 30 03:29:38.782352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.782311    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" event={"ID":"328703a5-2224-4370-a53c-fb9b59915ab2","Type":"ContainerDied","Data":"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"}
Oct 30 03:29:38.782445 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.782354    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl" event={"ID":"328703a5-2224-4370-a53c-fb9b59915ab2","Type":"ContainerDied","Data":"9952da68f586cc3f2b066fa698da6560d207201a4ab8d6133545908db2383fa7"}
Oct 30 03:29:38.791965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.791941    4214 scope.go:122] "RemoveContainer" containerID="6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"
Oct 30 03:29:38.792294 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:38.792272    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059\": container with ID starting with 6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059 not found: ID does not exist" containerID="6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"
Oct 30 03:29:38.792387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.792302    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059"} err="failed to get container status \"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059\": rpc error: code = NotFound desc = could not find container \"6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059\": container with ID starting with 6d39502f9ba40b0033cc87a7886a4da1c2f2a6c5ea54d8a028ced21efb193059 not found: ID does not exist"
Oct 30 03:29:38.792387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.792319    4214 scope.go:122] "RemoveContainer" containerID="ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"
Oct 30 03:29:38.802295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.802276    4214 scope.go:122] "RemoveContainer" containerID="ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"
Oct 30 03:29:38.802557 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:38.802539    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1\": container with ID starting with ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1 not found: ID does not exist" containerID="ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"
Oct 30 03:29:38.802648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.802566    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1"} err="failed to get container status \"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1\": rpc error: code = NotFound desc = could not find container \"ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1\": container with ID starting with ce834915b66dcb311f6efb376c34dbc4a73686de7fff09a0f78721935d242be1 not found: ID does not exist"
Oct 30 03:29:38.802648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.802580    4214 scope.go:122] "RemoveContainer" containerID="f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"
Oct 30 03:29:38.811357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.811337    4214 scope.go:122] "RemoveContainer" containerID="f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"
Oct 30 03:29:38.811625 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:38.811606    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d\": container with ID starting with f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d not found: ID does not exist" containerID="f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"
Oct 30 03:29:38.811692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.811631    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d"} err="failed to get container status \"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d\": rpc error: code = NotFound desc = could not find container \"f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d\": container with ID starting with f340c45bcb9452626d48e9cc0f28e3695b2f25d4d20d9660f299863cf59afa3d not found: ID does not exist"
Oct 30 03:29:38.826819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.826794    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlz4\" (UniqueName: \"kubernetes.io/projected/b830bf8a-554a-4597-ad34-9785136e7044-kube-api-access-7xlz4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.826819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.826817    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b830bf8a-554a-4597-ad34-9785136e7044-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:38.864675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.864638    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 03:29:38.867982 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.867957    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-32bad3-56784c8f76-7s7pl"]
Oct 30 03:29:38.902760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.902729    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 03:29:38.906316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.906290    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-32bad3-d7ffdf995-9qz7m"]
Oct 30 03:29:38.943119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.943094    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:29:38.947252 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:38.947200    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6b5f19-6b74866665-9cdfl"]
Oct 30 03:29:39.067366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:39.067333    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328703a5-2224-4370-a53c-fb9b59915ab2" path="/var/lib/kubelet/pods/328703a5-2224-4370-a53c-fb9b59915ab2/volumes"
Oct 30 03:29:39.067829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:39.067813    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" path="/var/lib/kubelet/pods/7ba075b4-a86a-4c49-adfd-5063f81a084e/volumes"
Oct 30 03:29:39.068191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:39.068176    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b830bf8a-554a-4597-ad34-9785136e7044" path="/var/lib/kubelet/pods/b830bf8a-554a-4597-ad34-9785136e7044/volumes"
Oct 30 03:29:39.165464 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:39.165426    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:29:44.414655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:44.414566    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:46.517915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:46.517865    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:46.552560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:46.552521    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:29:46.552776 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:46.552747    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" containerID="cri-o://ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a" gracePeriod=30
Oct 30 03:29:51.848578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.848547    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:29:51.900389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.900080    4214 generic.go:356] "Generic (PLEG): container finished" podID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerID="ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a" exitCode=0
Oct 30 03:29:51.900389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.900148    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"
Oct 30 03:29:51.900389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.900155    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerDied","Data":"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"}
Oct 30 03:29:51.900389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.900211    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" event={"ID":"02ad089a-84e3-4867-81c4-34890dc9e56e","Type":"ContainerDied","Data":"14d03b7a519b70caf693bde5907956e055600134b296dbb4e7b896f613a20396"}
Oct 30 03:29:51.900389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.900274    4214 scope.go:122] "RemoveContainer" containerID="ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"
Oct 30 03:29:51.911255 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.911221    4214 scope.go:122] "RemoveContainer" containerID="e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"
Oct 30 03:29:51.930546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.930519    4214 scope.go:122] "RemoveContainer" containerID="ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"
Oct 30 03:29:51.930974 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:51.930944    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a\": container with ID starting with ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a not found: ID does not exist" containerID="ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"
Oct 30 03:29:51.931092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.930992    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a"} err="failed to get container status \"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a\": rpc error: code = NotFound desc = could not find container \"ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a\": container with ID starting with ef6eacf066faf98d29fcec96e104fc25e16a77e9a57636ed030c9178dd8f634a not found: ID does not exist"
Oct 30 03:29:51.931092 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.931017    4214 scope.go:122] "RemoveContainer" containerID="e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"
Oct 30 03:29:51.931451 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:51.931421    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e\": container with ID starting with e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e not found: ID does not exist" containerID="e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"
Oct 30 03:29:51.931667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.931637    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e"} err="failed to get container status \"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e\": rpc error: code = NotFound desc = could not find container \"e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e\": container with ID starting with e5142258be0b0c7c4c5fb6c3d7d1cdd0cbf65f40b9047cc33efdc07d0cddab0e not found: ID does not exist"
Oct 30 03:29:51.987451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987413    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987475    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987519    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987568    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987594    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987673    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987717    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.987872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987799    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.988045 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.987877    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\") pod \"02ad089a-84e3-4867-81c4-34890dc9e56e\" (UID: \"02ad089a-84e3-4867-81c4-34890dc9e56e\") "
Oct 30 03:29:51.988519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.988493    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:51.989012 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.988999    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:51.989139 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.989000    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:51.989566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.989349    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:51.989944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.989669    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:51.990349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.990328    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:51.990879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.990842    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:51.991501 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.991477    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:51.992062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.992040    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "kube-api-access-skjm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:51.992489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:51.992469    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token" pod "02ad089a-84e3-4867-81c4-34890dc9e56e" (UID: "02ad089a-84e3-4867-81c4-34890dc9e56e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090253    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090282    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ad089a-84e3-4867-81c4-34890dc9e56e-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090292    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090301    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090309    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090317    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ad089a-84e3-4867-81c4-34890dc9e56e-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090325    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skjm6\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-kube-api-access-skjm6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.090334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.090334    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ad089a-84e3-4867-81c4-34890dc9e56e-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.253266 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.253213    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:29:52.255336 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.255301    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d"]
Oct 30 03:29:52.773301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.773261    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:29:52.773504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:52.773335    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-q4x8d" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:29:53.065993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:53.065961    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" path="/var/lib/kubelet/pods/02ad089a-84e3-4867-81c4-34890dc9e56e/volumes"
Oct 30 03:29:54.414957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:54.414913    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:56.518463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:56.518425    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.944135    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.945134    4214 generic.go:356] "Generic (PLEG): container finished" podID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerID="418db4de87babcdf886cac8cc71248faf2bdbf3ed9def61127b5a26754ba3ba3" exitCode=0
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.945259    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" event={"ID":"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e","Type":"ContainerDied","Data":"418db4de87babcdf886cac8cc71248faf2bdbf3ed9def61127b5a26754ba3ba3"}
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.946670    4214 generic.go:356] "Generic (PLEG): container finished" podID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerID="77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31" exitCode=0
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.946733    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.946734    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" event={"ID":"e0620af5-4d12-41b6-8528-5b51cea0f3f0","Type":"ContainerDied","Data":"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"}
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.946852    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d" event={"ID":"e0620af5-4d12-41b6-8528-5b51cea0f3f0","Type":"ContainerDied","Data":"6abd7c6faed3e6f77ced5e2b76d40bc4e5dcf098427171373405f2e794507433"}
Oct 30 03:29:57.950269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.946875    4214 scope.go:122] "RemoveContainer" containerID="77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"
Oct 30 03:29:57.955246 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.955205    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\" (UniqueName: \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\") pod \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") "
Oct 30 03:29:57.955447 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.955269    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\") pod \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\" (UID: \"e0620af5-4d12-41b6-8528-5b51cea0f3f0\") "
Oct 30 03:29:57.956275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.956219    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume" pod "e0620af5-4d12-41b6-8528-5b51cea0f3f0" (UID: "e0620af5-4d12-41b6-8528-5b51cea0f3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:57.957870 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.957841    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9" pod "e0620af5-4d12-41b6-8528-5b51cea0f3f0" (UID: "e0620af5-4d12-41b6-8528-5b51cea0f3f0"). InnerVolumeSpecName "kube-api-access-wdpq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:57.958570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.958552    4214 scope.go:122] "RemoveContainer" containerID="77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"
Oct 30 03:29:57.958886 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:29:57.958864    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31\": container with ID starting with 77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31 not found: ID does not exist" containerID="77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"
Oct 30 03:29:57.958964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.958896    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31"} err="failed to get container status \"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31\": rpc error: code = NotFound desc = could not find container \"77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31\": container with ID starting with 77d5ab47d4e75d752be3dd48789f0a5c87201cabbffa3847c7c50ad2e684cd31 not found: ID does not exist"
Oct 30 03:29:57.976169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:57.976145    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:29:58.057171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057139    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\") pod \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") "
Oct 30 03:29:58.057407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057315    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\" (UniqueName: \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\") pod \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\" (UID: \"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e\") "
Oct 30 03:29:58.057634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057605    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume" pod "7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" (UID: "7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:58.057807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057785    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:58.057885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057814    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpq9\" (UniqueName: \"kubernetes.io/projected/e0620af5-4d12-41b6-8528-5b51cea0f3f0-kube-api-access-wdpq9\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:58.057885 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.057828    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0620af5-4d12-41b6-8528-5b51cea0f3f0-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:58.059961 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.059934    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95" pod "7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" (UID: "7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e"). InnerVolumeSpecName "kube-api-access-vlx95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:58.160552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.159757    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlx95\" (UniqueName: \"kubernetes.io/projected/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e-kube-api-access-vlx95\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:29:58.297799 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.297765    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:29:58.310400 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.310373    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6b5f19-cbc474456-lgj2d"]
Oct 30 03:29:58.956853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.956757    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"
Oct 30 03:29:58.956853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.956776    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9" event={"ID":"7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e","Type":"ContainerDied","Data":"60d002fca1a056dff6813c4c4ffa41407730ac2c4f3c1dfaaa1834f4b2e62fed"}
Oct 30 03:29:58.956853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:58.956846    4214 scope.go:122] "RemoveContainer" containerID="418db4de87babcdf886cac8cc71248faf2bdbf3ed9def61127b5a26754ba3ba3"
Oct 30 03:29:59.009416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:59.009387    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:29:59.022576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:59.022552    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6b5f19-6ddd55fb5f-djmj9"]
Oct 30 03:29:59.069491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:59.069449    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" path="/var/lib/kubelet/pods/7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e/volumes"
Oct 30 03:29:59.070032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:29:59.070010    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" path="/var/lib/kubelet/pods/e0620af5-4d12-41b6-8528-5b51cea0f3f0/volumes"
Oct 30 03:30:00.181069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181035    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:00.181841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181821    4214 cpu_manager.go:414] "containerMap: removing container" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181842    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181854    4214 cpu_manager.go:414] "containerMap: removing container" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181863    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181880    4214 cpu_manager.go:414] "containerMap: removing container" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181888    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181916    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing"
Oct 30 03:30:00.181933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181927    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181938    4214 cpu_manager.go:414] "containerMap: removing container" podUID="328703a5-2224-4370-a53c-fb9b59915ab2" containerName="maas-api"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181945    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328703a5-2224-4370-a53c-fb9b59915ab2" containerName="maas-api"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181957    4214 cpu_manager.go:414] "containerMap: removing container" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181964    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181985    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.181992    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182025    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182033    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182045    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182051    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182153    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba075b4-a86a-4c49-adfd-5063f81a084e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182169    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7454b33a-0272-4cad-a8f2-8d2b1a939917" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182181    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e89013a-7eba-4c7a-bb9e-46d8dd72cb4e" containerName="payload-pre-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182191    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182200    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0620af5-4d12-41b6-8528-5b51cea0f3f0" containerName="payload-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182210    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a18368-7423-47c1-b0dc-c2aaf3999a42" containerName="main"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182218    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182241    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182253    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b830bf8a-554a-4597-ad34-9785136e7044" containerName="payload-processing"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182262    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="328703a5-2224-4370-a53c-fb9b59915ab2" containerName="maas-api"
Oct 30 03:30:00.182330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.182269    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ad089a-84e3-4867-81c4-34890dc9e56e" containerName="istio-proxy"
Oct 30 03:30:00.187611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.187584    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.204997 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.204974    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:00.278571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.278534    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:30:00.293065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.293034    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5d6\" (UniqueName: \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.293221 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.293100    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.393836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.393800    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5d6\" (UniqueName: \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.394027 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.393870    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.394581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.394560    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.415289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.415258    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5d6\" (UniqueName: \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.505771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.505692    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:00.816426 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.816394    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:00.817264 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:30:00.817211    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c415a63_7b04_4ee7_be1a_817c1a465fa3.slice/crio-55550881164a29c87fa26b6a0096d147bc9d12b3177556f2087511fa3c92cfaa WatchSource:0}: Error finding container 55550881164a29c87fa26b6a0096d147bc9d12b3177556f2087511fa3c92cfaa: Status 404 returned error can't find the container with id 55550881164a29c87fa26b6a0096d147bc9d12b3177556f2087511fa3c92cfaa
Oct 30 03:30:00.976109 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:00.976075    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" event={"ID":"9c415a63-7b04-4ee7-be1a-817c1a465fa3","Type":"ContainerStarted","Data":"55550881164a29c87fa26b6a0096d147bc9d12b3177556f2087511fa3c92cfaa"}
Oct 30 03:30:01.986887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:01.986831    4214 generic.go:356] "Generic (PLEG): container finished" podID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerID="19f471a18763f6c0c7afeb4fece407bbac7e01828dbcf2de5448c810a659c592" exitCode=60
Oct 30 03:30:01.987305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:01.986870    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" event={"ID":"9c415a63-7b04-4ee7-be1a-817c1a465fa3","Type":"ContainerDied","Data":"19f471a18763f6c0c7afeb4fece407bbac7e01828dbcf2de5448c810a659c592"}
Oct 30 03:30:01.987305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:01.987216    4214 scope.go:122] "RemoveContainer" containerID="19f471a18763f6c0c7afeb4fece407bbac7e01828dbcf2de5448c810a659c592"
Oct 30 03:30:02.995637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:02.995599    4214 generic.go:356] "Generic (PLEG): container finished" podID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerID="084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a" exitCode=60
Oct 30 03:30:02.996074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:02.995678    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" event={"ID":"9c415a63-7b04-4ee7-be1a-817c1a465fa3","Type":"ContainerDied","Data":"084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a"}
Oct 30 03:30:02.996074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:02.995738    4214 scope.go:122] "RemoveContainer" containerID="19f471a18763f6c0c7afeb4fece407bbac7e01828dbcf2de5448c810a659c592"
Oct 30 03:30:02.996074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:02.995981    4214 scope.go:122] "RemoveContainer" containerID="084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a"
Oct 30 03:30:02.996339 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:02.996318    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9_odh-ai-gateway-infra(9c415a63-7b04-4ee7-be1a-817c1a465fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3"
Oct 30 03:30:04.003610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:04.003583    4214 scope.go:122] "RemoveContainer" containerID="084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a"
Oct 30 03:30:04.004068 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:04.003863    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9_odh-ai-gateway-infra(9c415a63-7b04-4ee7-be1a-817c1a465fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3"
Oct 30 03:30:08.132095 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.132061    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:30:08.132595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.132310    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" podUID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" containerName="maas-api" containerID="cri-o://989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e" gracePeriod=30
Oct 30 03:30:08.434715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.434684    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:30:08.588190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588149    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\" (UniqueName: \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\") pod \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") "
Oct 30 03:30:08.588421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588242    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\") pod \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") "
Oct 30 03:30:08.588421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588345    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\") pod \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") "
Oct 30 03:30:08.588421 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588413    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\") pod \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\" (UID: \"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50\") "
Oct 30 03:30:08.588641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588610    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca" pod "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" (UID: "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:08.588917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.588881    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.593099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.593057    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls" pod "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" (UID: "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:08.593099 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.593073    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7" pod "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" (UID: "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50"). InnerVolumeSpecName "kube-api-access-474c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:08.602654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.602624    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs" pod "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" (UID: "26f0cdb8-ec94-4a1f-a68e-80ee33db0a50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:08.689774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.689678    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-474c7\" (UniqueName: \"kubernetes.io/projected/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-kube-api-access-474c7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.689774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.689716    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.689774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:08.689730    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:09.041367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.041263    4214 generic.go:356] "Generic (PLEG): container finished" podID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" containerID="989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e" exitCode=0
Oct 30 03:30:09.041367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.041316    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"
Oct 30 03:30:09.041367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.041335    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" event={"ID":"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50","Type":"ContainerDied","Data":"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"}
Oct 30 03:30:09.041661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.041391    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4" event={"ID":"26f0cdb8-ec94-4a1f-a68e-80ee33db0a50","Type":"ContainerDied","Data":"a4a1df38b6e749d2d93e5b664a15d7cfb7542a67a315cab58f14b5d57672d45e"}
Oct 30 03:30:09.041661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.041421    4214 scope.go:122] "RemoveContainer" containerID="989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"
Oct 30 03:30:09.053691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.053662    4214 scope.go:122] "RemoveContainer" containerID="989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"
Oct 30 03:30:09.054098 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:09.054069    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e\": container with ID starting with 989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e not found: ID does not exist" containerID="989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"
Oct 30 03:30:09.054168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.054112    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e"} err="failed to get container status \"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e\": rpc error: code = NotFound desc = could not find container \"989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e\": container with ID starting with 989d68629b17452670659351ce52861cabc26ce313307577ba44bb2be7048c6e not found: ID does not exist"
Oct 30 03:30:09.096220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.096176    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:30:09.108821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:09.108783    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ae1dd9-56c7d4695c-lrcs4"]
Oct 30 03:30:09.166739 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:09.166682    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:30:11.066288 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:11.066252    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" path="/var/lib/kubelet/pods/26f0cdb8-ec94-4a1f-a68e-80ee33db0a50/volumes"
Oct 30 03:30:18.059239 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:18.059194    4214 scope.go:122] "RemoveContainer" containerID="084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a"
Oct 30 03:30:19.120670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:19.120197    4214 generic.go:356] "Generic (PLEG): container finished" podID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerID="f035fb2e55619ea71526dadaac4dd283ee33ee6f2d9938a168f63c864df5c49f" exitCode=60
Oct 30 03:30:19.120670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:19.120263    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" event={"ID":"9c415a63-7b04-4ee7-be1a-817c1a465fa3","Type":"ContainerDied","Data":"f035fb2e55619ea71526dadaac4dd283ee33ee6f2d9938a168f63c864df5c49f"}
Oct 30 03:30:19.120670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:19.120332    4214 scope.go:122] "RemoveContainer" containerID="084f1b2a3694ec57eb25833b49ed9071b31efc9541b505ff467aa9cfaa43fb7a"
Oct 30 03:30:19.121242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:19.120772    4214 scope.go:122] "RemoveContainer" containerID="f035fb2e55619ea71526dadaac4dd283ee33ee6f2d9938a168f63c864df5c49f"
Oct 30 03:30:19.121242 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:19.121047    4214 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9_odh-ai-gateway-infra(9c415a63-7b04-4ee7-be1a-817c1a465fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3"
Oct 30 03:30:20.170579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.170539    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:20.340002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.339976    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:20.427842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.427767    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\" (UniqueName: \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\") pod \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") "
Oct 30 03:30:20.427842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.427816    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\") pod \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\" (UID: \"9c415a63-7b04-4ee7-be1a-817c1a465fa3\") "
Oct 30 03:30:20.428271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.428220    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca" pod "9c415a63-7b04-4ee7-be1a-817c1a465fa3" (UID: "9c415a63-7b04-4ee7-be1a-817c1a465fa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.429993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.429966    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6" pod "9c415a63-7b04-4ee7-be1a-817c1a465fa3" (UID: "9c415a63-7b04-4ee7-be1a-817c1a465fa3"). InnerVolumeSpecName "kube-api-access-dw5d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.529073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.529042    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5d6\" (UniqueName: \"kubernetes.io/projected/9c415a63-7b04-4ee7-be1a-817c1a465fa3-kube-api-access-dw5d6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.529073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:20.529070    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c415a63-7b04-4ee7-be1a-817c1a465fa3-maas-api-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.137340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:21.137303    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"
Oct 30 03:30:21.137527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:21.137307    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9" event={"ID":"9c415a63-7b04-4ee7-be1a-817c1a465fa3","Type":"ContainerDied","Data":"55550881164a29c87fa26b6a0096d147bc9d12b3177556f2087511fa3c92cfaa"}
Oct 30 03:30:21.137527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:21.137493    4214 scope.go:122] "RemoveContainer" containerID="f035fb2e55619ea71526dadaac4dd283ee33ee6f2d9938a168f63c864df5c49f"
Oct 30 03:30:21.192676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:21.192641    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:21.198361 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:21.198333    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-874ab5-29888850-29rj9"]
Oct 30 03:30:23.066514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:23.066485    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" path="/var/lib/kubelet/pods/9c415a63-7b04-4ee7-be1a-817c1a465fa3/volumes"
Oct 30 03:30:26.585413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:26.585378    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 03:30:26.622003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:26.621966    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 03:30:30.574781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.574740    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:30:30.575521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.575009    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" podUID="1a152e51-e5bf-426d-9e44-4b4c862971e1" containerName="maas-api" containerID="cri-o://0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c" gracePeriod=30
Oct 30 03:30:30.877763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.877735    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:30:30.943073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943035    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\") pod \"1a152e51-e5bf-426d-9e44-4b4c862971e1\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") "
Oct 30 03:30:30.943326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943136    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\") pod \"1a152e51-e5bf-426d-9e44-4b4c862971e1\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") "
Oct 30 03:30:30.943326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943204    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\") pod \"1a152e51-e5bf-426d-9e44-4b4c862971e1\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") "
Oct 30 03:30:30.943326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943293    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\" (UniqueName: \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\") pod \"1a152e51-e5bf-426d-9e44-4b4c862971e1\" (UID: \"1a152e51-e5bf-426d-9e44-4b4c862971e1\") "
Oct 30 03:30:30.943594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943565    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca" pod "1a152e51-e5bf-426d-9e44-4b4c862971e1" (UID: "1a152e51-e5bf-426d-9e44-4b4c862971e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:30.943759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.943736    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a152e51-e5bf-426d-9e44-4b4c862971e1-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:30.946270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.945702    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs" pod "1a152e51-e5bf-426d-9e44-4b4c862971e1" (UID: "1a152e51-e5bf-426d-9e44-4b4c862971e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:30.946270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.946063    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls" pod "1a152e51-e5bf-426d-9e44-4b4c862971e1" (UID: "1a152e51-e5bf-426d-9e44-4b4c862971e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:30.960619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:30.960584    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv" pod "1a152e51-e5bf-426d-9e44-4b4c862971e1" (UID: "1a152e51-e5bf-426d-9e44-4b4c862971e1"). InnerVolumeSpecName "kube-api-access-4sslv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:31.051262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.045170    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.051262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.045213    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a152e51-e5bf-426d-9e44-4b4c862971e1-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.051262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.045243    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sslv\" (UniqueName: \"kubernetes.io/projected/1a152e51-e5bf-426d-9e44-4b4c862971e1-kube-api-access-4sslv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.206942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.206840    4214 generic.go:356] "Generic (PLEG): container finished" podID="1a152e51-e5bf-426d-9e44-4b4c862971e1" containerID="0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c" exitCode=0
Oct 30 03:30:31.206942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.206905    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"
Oct 30 03:30:31.206942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.206924    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" event={"ID":"1a152e51-e5bf-426d-9e44-4b4c862971e1","Type":"ContainerDied","Data":"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"}
Oct 30 03:30:31.207431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.207006    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586" event={"ID":"1a152e51-e5bf-426d-9e44-4b4c862971e1","Type":"ContainerDied","Data":"0b8c0544d8ff39a10d528bd7163039099a8aa843a3f7deb408a8151459e40605"}
Oct 30 03:30:31.207431 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.207046    4214 scope.go:122] "RemoveContainer" containerID="0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"
Oct 30 03:30:31.218157 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.218132    4214 scope.go:122] "RemoveContainer" containerID="0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"
Oct 30 03:30:31.218592 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:31.218565    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c\": container with ID starting with 0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c not found: ID does not exist" containerID="0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"
Oct 30 03:30:31.218726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.218597    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c"} err="failed to get container status \"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c\": rpc error: code = NotFound desc = could not find container \"0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c\": container with ID starting with 0780b8dddc4f7eaaa2d2c27d659ac1c676ffdbdbc173b9fcbf119b3b74d01e8c not found: ID does not exist"
Oct 30 03:30:31.264962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.264905    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:30:31.268839 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.268810    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf605a-845dc5687b-cb586"]
Oct 30 03:30:31.586710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.586667    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing" containerID="cri-o://bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4" gracePeriod=25
Oct 30 03:30:31.623318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:31.623276    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing" containerID="cri-o://00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1" gracePeriod=25
Oct 30 03:30:33.067511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:33.067467    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a152e51-e5bf-426d-9e44-4b4c862971e1" path="/var/lib/kubelet/pods/1a152e51-e5bf-426d-9e44-4b4c862971e1/volumes"
Oct 30 03:30:35.501887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:35.501852    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:36.512725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:36.512685    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:36.725076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:36.725036    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:30:39.163682 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:39.163641    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:30:45.502066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:45.502027    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:46.513050 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:46.513011    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:51.748309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.748269    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" podUID="6a3e0c57-eed0-4005-99f5-3c03171418d2" containerName="main" containerID="cri-o://3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3" gracePeriod=45
Oct 30 03:30:51.910589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.910558    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 03:30:51.931980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.931951    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 03:30:51.978630 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.978597    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\") pod \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") "
Oct 30 03:30:51.978948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.978931    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\" (UniqueName: \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\") pod \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\" (UID: \"41fb0d90-45b4-454f-95f3-ae7c84e2448f\") "
Oct 30 03:30:51.979125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.978984    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume" pod "41fb0d90-45b4-454f-95f3-ae7c84e2448f" (UID: "41fb0d90-45b4-454f-95f3-ae7c84e2448f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:51.979702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.979679    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41fb0d90-45b4-454f-95f3-ae7c84e2448f-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.981628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:51.981599    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp" pod "41fb0d90-45b4-454f-95f3-ae7c84e2448f" (UID: "41fb0d90-45b4-454f-95f3-ae7c84e2448f"). InnerVolumeSpecName "kube-api-access-tdspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:52.019993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.019968    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:30:52.081026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.080973    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\") pod \"61881be2-60e7-480b-8ab8-40544091d90e\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") "
Oct 30 03:30:52.081222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.081173    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\" (UniqueName: \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\") pod \"61881be2-60e7-480b-8ab8-40544091d90e\" (UID: \"61881be2-60e7-480b-8ab8-40544091d90e\") "
Oct 30 03:30:52.081423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.081400    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume" pod "61881be2-60e7-480b-8ab8-40544091d90e" (UID: "61881be2-60e7-480b-8ab8-40544091d90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:52.081615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.081598    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdspp\" (UniqueName: \"kubernetes.io/projected/41fb0d90-45b4-454f-95f3-ae7c84e2448f-kube-api-access-tdspp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.081668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.081622    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61881be2-60e7-480b-8ab8-40544091d90e-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.083365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.083341    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89" pod "61881be2-60e7-480b-8ab8-40544091d90e" (UID: "61881be2-60e7-480b-8ab8-40544091d90e"). InnerVolumeSpecName "kube-api-access-w2f89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:52.182573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182481    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\" (UniqueName: \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.182759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182577    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.182759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182642    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.182759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182695    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.182759 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182728    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.183015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182764    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\") pod \"6a3e0c57-eed0-4005-99f5-3c03171418d2\" (UID: \"6a3e0c57-eed0-4005-99f5-3c03171418d2\") "
Oct 30 03:30:52.183015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182911    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:52.183128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.182993    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:52.183128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.183051    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:52.183314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.183292    4214 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-tmp-dir\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.183368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.183319    4214 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-model-cache\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.183368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.183334    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f89\" (UniqueName: \"kubernetes.io/projected/61881be2-60e7-480b-8ab8-40544091d90e-kube-api-access-w2f89\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.183368 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.183350    4214 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-home\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.184836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.184810    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:52.184904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.184861    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:52.184904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.184876    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2" pod "6a3e0c57-eed0-4005-99f5-3c03171418d2" (UID: "6a3e0c57-eed0-4005-99f5-3c03171418d2"). InnerVolumeSpecName "kube-api-access-4fzb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:52.284732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.284693    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzb2\" (UniqueName: \"kubernetes.io/projected/6a3e0c57-eed0-4005-99f5-3c03171418d2-kube-api-access-4fzb2\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.284732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.284725    4214 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3e0c57-eed0-4005-99f5-3c03171418d2-tls-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.284732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.284735    4214 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3e0c57-eed0-4005-99f5-3c03171418d2-dshm\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.340051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.340014    4214 generic.go:356] "Generic (PLEG): container finished" podID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerID="bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4" exitCode=0
Oct 30 03:30:52.340299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.340057    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"
Oct 30 03:30:52.340299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.340095    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" event={"ID":"41fb0d90-45b4-454f-95f3-ae7c84e2448f","Type":"ContainerDied","Data":"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"}
Oct 30 03:30:52.340299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.340150    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9" event={"ID":"41fb0d90-45b4-454f-95f3-ae7c84e2448f","Type":"ContainerDied","Data":"3e4e9697994f63277e123750dc9079510236d39d50433fb016de525e9915f419"}
Oct 30 03:30:52.340299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.340179    4214 scope.go:122] "RemoveContainer" containerID="bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"
Oct 30 03:30:52.341808 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.341784    4214 generic.go:356] "Generic (PLEG): container finished" podID="6a3e0c57-eed0-4005-99f5-3c03171418d2" containerID="3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3" exitCode=0
Oct 30 03:30:52.341917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.341837    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"
Oct 30 03:30:52.342022 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.341865    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" event={"ID":"6a3e0c57-eed0-4005-99f5-3c03171418d2","Type":"ContainerDied","Data":"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"}
Oct 30 03:30:52.342104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.342032    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452" event={"ID":"6a3e0c57-eed0-4005-99f5-3c03171418d2","Type":"ContainerDied","Data":"91bdc3037d11c658d36d9f140bc6587913e823b6d4b7f8af42f81b3fa3890820"}
Oct 30 03:30:52.343737 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.343712    4214 generic.go:356] "Generic (PLEG): container finished" podID="61881be2-60e7-480b-8ab8-40544091d90e" containerID="00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1" exitCode=0
Oct 30 03:30:52.343853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.343750    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" event={"ID":"61881be2-60e7-480b-8ab8-40544091d90e","Type":"ContainerDied","Data":"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"}
Oct 30 03:30:52.343853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.343784    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z" event={"ID":"61881be2-60e7-480b-8ab8-40544091d90e","Type":"ContainerDied","Data":"26054591336b01faee737e70727a61b83d405376e229c08e7660879ad97fad41"}
Oct 30 03:30:52.343853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.343754    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"
Oct 30 03:30:52.352888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.352827    4214 scope.go:122] "RemoveContainer" containerID="bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"
Oct 30 03:30:52.353162 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:52.353139    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4\": container with ID starting with bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4 not found: ID does not exist" containerID="bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"
Oct 30 03:30:52.353271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.353174    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4"} err="failed to get container status \"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4\": rpc error: code = NotFound desc = could not find container \"bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4\": container with ID starting with bc91eae4b88eccdea975ff9c509b5025d72579e2b9eb64ac41b48bbd6db1fdd4 not found: ID does not exist"
Oct 30 03:30:52.353271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.353197    4214 scope.go:122] "RemoveContainer" containerID="3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"
Oct 30 03:30:52.363855 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.363838    4214 scope.go:122] "RemoveContainer" containerID="3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"
Oct 30 03:30:52.364168 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:52.364147    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3\": container with ID starting with 3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3 not found: ID does not exist" containerID="3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"
Oct 30 03:30:52.364270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.364180    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3"} err="failed to get container status \"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3\": rpc error: code = NotFound desc = could not find container \"3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3\": container with ID starting with 3d87dfe11166033481f57d79f82060dde01757818dcf2a730963a70c2b9263d3 not found: ID does not exist"
Oct 30 03:30:52.364270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.364202    4214 scope.go:122] "RemoveContainer" containerID="00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"
Oct 30 03:30:52.373604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.373579    4214 scope.go:122] "RemoveContainer" containerID="00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"
Oct 30 03:30:52.373860 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:52.373840    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1\": container with ID starting with 00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1 not found: ID does not exist" containerID="00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"
Oct 30 03:30:52.373928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.373872    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1"} err="failed to get container status \"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1\": rpc error: code = NotFound desc = could not find container \"00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1\": container with ID starting with 00349e23a3deb9fadf7e7c69024fbc57e9f1cd4a597f8986b37a80b8da0c86a1 not found: ID does not exist"
Oct 30 03:30:52.425464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.425431    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 03:30:52.438075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.437992    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7d1a1-7dcbd5df7b-clmq9"]
Oct 30 03:30:52.465200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.465171    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:30:52.468316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.468291    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cf605a/ipp-route-cf605a-kserve-78779c7665-sj452"]
Oct 30 03:30:52.503344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.503312    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 03:30:52.506923 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:52.506887    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7d1a1-55d6db6f9f-9dd5z"]
Oct 30 03:30:53.043104 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.043069    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 03:30:53.043594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.043319    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" podUID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" containerName="maas-api" containerID="cri-o://dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c" gracePeriod=30
Oct 30 03:30:53.083373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.079600    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" path="/var/lib/kubelet/pods/41fb0d90-45b4-454f-95f3-ae7c84e2448f/volumes"
Oct 30 03:30:53.083373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.080196    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61881be2-60e7-480b-8ab8-40544091d90e" path="/var/lib/kubelet/pods/61881be2-60e7-480b-8ab8-40544091d90e/volumes"
Oct 30 03:30:53.083373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.080770    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3e0c57-eed0-4005-99f5-3c03171418d2" path="/var/lib/kubelet/pods/6a3e0c57-eed0-4005-99f5-3c03171418d2/volumes"
Oct 30 03:30:53.322849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.322813    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 03:30:53.352472 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.352437    4214 generic.go:356] "Generic (PLEG): container finished" podID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" containerID="dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c" exitCode=0
Oct 30 03:30:53.352680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.352480    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"
Oct 30 03:30:53.352680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.352515    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" event={"ID":"c40f86a2-bcfa-41f4-a00c-433ebea44c71","Type":"ContainerDied","Data":"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"}
Oct 30 03:30:53.352680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.352562    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2" event={"ID":"c40f86a2-bcfa-41f4-a00c-433ebea44c71","Type":"ContainerDied","Data":"a34da35f540223e802a6ba9dbf2f39b28c09481c6c88bce0e8bd6a722ea28077"}
Oct 30 03:30:53.352680 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.352587    4214 scope.go:122] "RemoveContainer" containerID="dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"
Oct 30 03:30:53.362213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.362167    4214 scope.go:122] "RemoveContainer" containerID="dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"
Oct 30 03:30:53.362562 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:30:53.362527    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c\": container with ID starting with dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c not found: ID does not exist" containerID="dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"
Oct 30 03:30:53.362729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.362569    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c"} err="failed to get container status \"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c\": rpc error: code = NotFound desc = could not find container \"dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c\": container with ID starting with dcc0ea4b2ec399e858b5045adfbf1b87c53dd175374832d16e239b706105c66c not found: ID does not exist"
Oct 30 03:30:53.396857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.396821    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\") pod \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") "
Oct 30 03:30:53.397033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.396917    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\") pod \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") "
Oct 30 03:30:53.397033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.396994    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\") pod \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") "
Oct 30 03:30:53.397177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.397080    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\" (UniqueName: \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\") pod \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\" (UID: \"c40f86a2-bcfa-41f4-a00c-433ebea44c71\") "
Oct 30 03:30:53.397308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.397277    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca" pod "c40f86a2-bcfa-41f4-a00c-433ebea44c71" (UID: "c40f86a2-bcfa-41f4-a00c-433ebea44c71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:53.397586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.397555    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c40f86a2-bcfa-41f4-a00c-433ebea44c71-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:53.399429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.399393    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s" pod "c40f86a2-bcfa-41f4-a00c-433ebea44c71" (UID: "c40f86a2-bcfa-41f4-a00c-433ebea44c71"). InnerVolumeSpecName "kube-api-access-xxj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:53.399719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.399674    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls" pod "c40f86a2-bcfa-41f4-a00c-433ebea44c71" (UID: "c40f86a2-bcfa-41f4-a00c-433ebea44c71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:53.399852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.399754    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs" pod "c40f86a2-bcfa-41f4-a00c-433ebea44c71" (UID: "c40f86a2-bcfa-41f4-a00c-433ebea44c71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:53.503015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.502968    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:53.503015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.503002    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c40f86a2-bcfa-41f4-a00c-433ebea44c71-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:53.503015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.503025    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxj6s\" (UniqueName: \"kubernetes.io/projected/c40f86a2-bcfa-41f4-a00c-433ebea44c71-kube-api-access-xxj6s\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:30:53.704750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.704060    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 03:30:53.710261 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:53.710207    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f7d1a1-5569b669c5-skgt2"]
Oct 30 03:30:55.065655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:30:55.065621    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" path="/var/lib/kubelet/pods/c40f86a2-bcfa-41f4-a00c-433ebea44c71/volumes"
Oct 30 03:31:07.508335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:07.508299    4214 scope.go:122] "RemoveContainer" containerID="e864afa9e2f9e9cb9eb805e30e30e6f6c4bdd7f43d4ceea47480b5b059babeb5"
Oct 30 03:31:09.165334 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:31:09.165296    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:31:20.315859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:20.315650    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:31:20.316497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:20.315917    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy" containerID="cri-o://6dafea288556d14c4ba27b8f87a3bd64ec9ac323291c0127f23b64cebba942f1" gracePeriod=30
Oct 30 03:31:25.555146 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.555106    4214 generic.go:356] "Generic (PLEG): container finished" podID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerID="6dafea288556d14c4ba27b8f87a3bd64ec9ac323291c0127f23b64cebba942f1" exitCode=0
Oct 30 03:31:25.555583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.555156    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerDied","Data":"6dafea288556d14c4ba27b8f87a3bd64ec9ac323291c0127f23b64cebba942f1"}
Oct 30 03:31:25.555583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.555220    4214 scope.go:122] "RemoveContainer" containerID="26b4d6f210fa446e37fc6b6134bbd5255589b044fec003e15eb02e8192361f3c"
Oct 30 03:31:25.593744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.593682    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:31:25.662958 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.662920    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.662989    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663085    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663137    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663197    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663244    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663322    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663365 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663324    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.663580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663376    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663406    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.663580 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663414    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\") pod \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\" (UID: \"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc\") "
Oct 30 03:31:25.663738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663696    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.663738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663712    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.663949 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663927    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.664021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663958    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.664021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663975    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.664021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.663990    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.664186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.664150    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:25.665725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.665697    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "kube-api-access-fzmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:25.665863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.665796    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.665935 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.665868    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:25.666907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.666876    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo" pod "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" (UID: "5bfa4bd8-5357-452a-80ea-0f694fe8e6fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:25.765473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.765434    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.765473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.765466    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.765473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.765475    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.765473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.765484    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.765755 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:25.765492    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmh7\" (UniqueName: \"kubernetes.io/projected/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc-kube-api-access-fzmh7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:26.562173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:26.562142    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"
Oct 30 03:31:26.562653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:26.562176    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg" event={"ID":"5bfa4bd8-5357-452a-80ea-0f694fe8e6fc","Type":"ContainerDied","Data":"2f9d73c0d4811e4e0b5ffa01cddf29a1a8f9ad75aa707594f20a5a10e4aae308"}
Oct 30 03:31:26.562653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:26.562245    4214 scope.go:122] "RemoveContainer" containerID="6dafea288556d14c4ba27b8f87a3bd64ec9ac323291c0127f23b64cebba942f1"
Oct 30 03:31:26.629287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:26.629222    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:31:26.635761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:26.635729    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cf605a-openshift-default-79f69d44f-xggxg"]
Oct 30 03:31:27.066194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:27.066151    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" path="/var/lib/kubelet/pods/5bfa4bd8-5357-452a-80ea-0f694fe8e6fc/volumes"
Oct 30 03:31:39.166968 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:31:39.166927    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:31:50.806315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:50.806276    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:31:50.835446 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:50.835413    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:31:53.355118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.355071    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:31:53.355748 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.355327    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" podUID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" containerName="maas-api" containerID="cri-o://ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285" gracePeriod=30
Oct 30 03:31:53.637024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.636991    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:31:53.734249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.734199    4214 generic.go:356] "Generic (PLEG): container finished" podID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" containerID="ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285" exitCode=0
Oct 30 03:31:53.734462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.734275    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" event={"ID":"559ed535-5e2e-4e43-9a1c-0b38dd3f2843","Type":"ContainerDied","Data":"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"}
Oct 30 03:31:53.734462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.734332    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f" event={"ID":"559ed535-5e2e-4e43-9a1c-0b38dd3f2843","Type":"ContainerDied","Data":"2aac41b92e6277a317ef753ab24e2cb524f446120a787e905ef33c4ad6494205"}
Oct 30 03:31:53.734462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.734356    4214 scope.go:122] "RemoveContainer" containerID="ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"
Oct 30 03:31:53.734462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.734284    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"
Oct 30 03:31:53.745074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.745057    4214 scope.go:122] "RemoveContainer" containerID="ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"
Oct 30 03:31:53.745388 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:31:53.745368    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285\": container with ID starting with ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285 not found: ID does not exist" containerID="ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"
Oct 30 03:31:53.745474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.745401    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285"} err="failed to get container status \"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285\": rpc error: code = NotFound desc = could not find container \"ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285\": container with ID starting with ae677ba7761d161004ffa6c6c35a9846bd0119c8b3aaab562fb86823cd096285 not found: ID does not exist"
Oct 30 03:31:53.748767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.748746    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") pod \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") "
Oct 30 03:31:53.748837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.748805    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\" (UniqueName: \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\") pod \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") "
Oct 30 03:31:53.748879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.748841    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\") pod \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") "
Oct 30 03:31:53.748968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.748952    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\") pod \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\" (UID: \"559ed535-5e2e-4e43-9a1c-0b38dd3f2843\") "
Oct 30 03:31:53.749306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.749220    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca" pod "559ed535-5e2e-4e43-9a1c-0b38dd3f2843" (UID: "559ed535-5e2e-4e43-9a1c-0b38dd3f2843"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:53.749442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.749402    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:53.751067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.751035    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls" pod "559ed535-5e2e-4e43-9a1c-0b38dd3f2843" (UID: "559ed535-5e2e-4e43-9a1c-0b38dd3f2843"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:53.751460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.751438    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs" pod "559ed535-5e2e-4e43-9a1c-0b38dd3f2843" (UID: "559ed535-5e2e-4e43-9a1c-0b38dd3f2843"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:53.751559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.751466    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8" pod "559ed535-5e2e-4e43-9a1c-0b38dd3f2843" (UID: "559ed535-5e2e-4e43-9a1c-0b38dd3f2843"). InnerVolumeSpecName "kube-api-access-jf2z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:53.850500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.850464    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:53.850500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.850496    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2z8\" (UniqueName: \"kubernetes.io/projected/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-kube-api-access-jf2z8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:53.850500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:53.850506    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559ed535-5e2e-4e43-9a1c-0b38dd3f2843-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:31:54.085520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:54.085482    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:31:54.087846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:54.087822    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-d823d1-6f8bdcd649-t425f"]
Oct 30 03:31:55.067396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:55.067363    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" path="/var/lib/kubelet/pods/559ed535-5e2e-4e43-9a1c-0b38dd3f2843/volumes"
Oct 30 03:31:55.806812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:55.806771    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing" containerID="cri-o://5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3" gracePeriod=25
Oct 30 03:31:55.836500 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:31:55.836421    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing" containerID="cri-o://561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b" gracePeriod=25
Oct 30 03:32:01.206550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:01.206512    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:32:01.206953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:01.206777    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" containerID="cri-o://0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a" gracePeriod=30
Oct 30 03:32:01.434350 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:01.434307    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:02.463480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:02.463439    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:03.661478 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:03.661438    4214 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:32:03.661888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:03.661522    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:32:06.462239 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.462205    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:32:06.581015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.580983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581038    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581059    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581110    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581133    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581279    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581313    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581420    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581428    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581446    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581458    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581500    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\") pod \"f592998c-cb32-4292-a4ac-86eca80e305e\" (UID: \"f592998c-cb32-4292-a4ac-86eca80e305e\") "
Oct 30 03:32:06.581827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581548    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:06.581827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581663    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.582015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.581991    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f592998c-cb32-4292-a4ac-86eca80e305e-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.582077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.582018    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.582077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.582034    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.582077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.582047    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.582077 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.582062    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.583348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.583317    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "kube-api-access-zsscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:06.583348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.583339    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.583910 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.583884    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:32:06.584010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.583907    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token" pod "f592998c-cb32-4292-a4ac-86eca80e305e" (UID: "f592998c-cb32-4292-a4ac-86eca80e305e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:06.682714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.682686    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f592998c-cb32-4292-a4ac-86eca80e305e-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.682714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.682712    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f592998c-cb32-4292-a4ac-86eca80e305e-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.682888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.682721    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.682888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.682730    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsscz\" (UniqueName: \"kubernetes.io/projected/f592998c-cb32-4292-a4ac-86eca80e305e-kube-api-access-zsscz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.819484 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.819449    4214 generic.go:356] "Generic (PLEG): container finished" podID="f592998c-cb32-4292-a4ac-86eca80e305e" containerID="0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a" exitCode=0
Oct 30 03:32:06.819656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.819504    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerDied","Data":"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"}
Oct 30 03:32:06.819656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.819521    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"
Oct 30 03:32:06.819656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.819579    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l" event={"ID":"f592998c-cb32-4292-a4ac-86eca80e305e","Type":"ContainerDied","Data":"b0bba3942f1d8f46d5fe6ac3134338c4a897fc9e4f87b0e124fce36bcaa008bf"}
Oct 30 03:32:06.819656 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.819602    4214 scope.go:122] "RemoveContainer" containerID="0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"
Oct 30 03:32:06.829677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.829651    4214 scope.go:122] "RemoveContainer" containerID="9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"
Oct 30 03:32:06.838820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.838802    4214 scope.go:122] "RemoveContainer" containerID="0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"
Oct 30 03:32:06.839083 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:06.839062    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a\": container with ID starting with 0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a not found: ID does not exist" containerID="0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"
Oct 30 03:32:06.839165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.839093    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a"} err="failed to get container status \"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a\": rpc error: code = NotFound desc = could not find container \"0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a\": container with ID starting with 0484de229c2634be829828557b5c6d6b9a68c5fa56ea71eb3fa4c229438c247a not found: ID does not exist"
Oct 30 03:32:06.839165 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.839117    4214 scope.go:122] "RemoveContainer" containerID="9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"
Oct 30 03:32:06.839367 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:06.839350    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c\": container with ID starting with 9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c not found: ID does not exist" containerID="9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"
Oct 30 03:32:06.839409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.839374    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c"} err="failed to get container status \"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c\": rpc error: code = NotFound desc = could not find container \"9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c\": container with ID starting with 9d3a8e6788d9a99ac5ecf3c80581cf2cccbda12bc4b3a1c548c85b74687e355c not found: ID does not exist"
Oct 30 03:32:06.872653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.872633    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:32:06.885222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:06.885202    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-tdn2l"]
Oct 30 03:32:07.066590 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:07.066551    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" path="/var/lib/kubelet/pods/f592998c-cb32-4292-a4ac-86eca80e305e/volumes"
Oct 30 03:32:09.163710 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:09.163673    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:32:11.435021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:11.434977    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:12.165613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.165574    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:32:12.192631 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.192592    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:32:12.463664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.463581    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:12.511578 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.511545    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:32:12.511818 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.511787    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" podUID="ede07f81-f152-4233-af36-5af83c59dec3" containerName="maas-api" containerID="cri-o://f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e" gracePeriod=30
Oct 30 03:32:12.567345 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.567309    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" podUID="ede07f81-f152-4233-af36-5af83c59dec3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.21:8443/health\": dial tcp 10.133.1.21:8443: connect: connection refused"
Oct 30 03:32:12.763554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.763523    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:32:12.857284 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.857166    4214 generic.go:356] "Generic (PLEG): container finished" podID="ede07f81-f152-4233-af36-5af83c59dec3" containerID="f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e" exitCode=0
Oct 30 03:32:12.857496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.857239    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"
Oct 30 03:32:12.857496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.857256    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" event={"ID":"ede07f81-f152-4233-af36-5af83c59dec3","Type":"ContainerDied","Data":"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"}
Oct 30 03:32:12.857496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.857475    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj" event={"ID":"ede07f81-f152-4233-af36-5af83c59dec3","Type":"ContainerDied","Data":"71c8811285eff4f731c34704e877d9690062ff429d660b3c8bcdcdcc9a806269"}
Oct 30 03:32:12.857690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.857514    4214 scope.go:122] "RemoveContainer" containerID="f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"
Oct 30 03:32:12.868110 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.868088    4214 scope.go:122] "RemoveContainer" containerID="f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"
Oct 30 03:32:12.868429 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:12.868395    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e\": container with ID starting with f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e not found: ID does not exist" containerID="f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"
Oct 30 03:32:12.868429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.868425    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e"} err="failed to get container status \"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e\": rpc error: code = NotFound desc = could not find container \"f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e\": container with ID starting with f130c7f98380bd6c08f95181321d44080d2f4ce0c1890e3e514431165ae7ad3e not found: ID does not exist"
Oct 30 03:32:12.939018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.938983    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\") pod \"ede07f81-f152-4233-af36-5af83c59dec3\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") "
Oct 30 03:32:12.939247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.939068    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\") pod \"ede07f81-f152-4233-af36-5af83c59dec3\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") "
Oct 30 03:32:12.939247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.939169    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\" (UniqueName: \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\") pod \"ede07f81-f152-4233-af36-5af83c59dec3\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") "
Oct 30 03:32:12.939247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.939199    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") pod \"ede07f81-f152-4233-af36-5af83c59dec3\" (UID: \"ede07f81-f152-4233-af36-5af83c59dec3\") "
Oct 30 03:32:12.939593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.939561    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca" pod "ede07f81-f152-4233-af36-5af83c59dec3" (UID: "ede07f81-f152-4233-af36-5af83c59dec3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:12.942023 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.941994    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs" pod "ede07f81-f152-4233-af36-5af83c59dec3" (UID: "ede07f81-f152-4233-af36-5af83c59dec3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:12.942363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.942335    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq" pod "ede07f81-f152-4233-af36-5af83c59dec3" (UID: "ede07f81-f152-4233-af36-5af83c59dec3"). InnerVolumeSpecName "kube-api-access-lzkpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:12.943314 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:12.943288    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls" pod "ede07f81-f152-4233-af36-5af83c59dec3" (UID: "ede07f81-f152-4233-af36-5af83c59dec3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:13.040821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.040726    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzkpq\" (UniqueName: \"kubernetes.io/projected/ede07f81-f152-4233-af36-5af83c59dec3-kube-api-access-lzkpq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:13.040821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.040757    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:13.040821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.040767    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede07f81-f152-4233-af36-5af83c59dec3-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:13.040821 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.040775    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede07f81-f152-4233-af36-5af83c59dec3-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:13.198796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.198759    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:32:13.200968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:13.200941    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-874ab5-598cdb6bf9-ghklj"]
Oct 30 03:32:15.066035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:15.066001    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede07f81-f152-4233-af36-5af83c59dec3" path="/var/lib/kubelet/pods/ede07f81-f152-4233-af36-5af83c59dec3/volumes"
Oct 30 03:32:16.107462 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.107434    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:32:16.161083 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.161050    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:32:16.277743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.277697    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\") pod \"e32f8695-0896-40d7-9893-ac55d5d161b0\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") "
Oct 30 03:32:16.277955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.277806    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\") pod \"9c57d49a-895e-4077-8c68-4b7637713d87\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") "
Oct 30 03:32:16.277955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.277876    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\" (UniqueName: \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\") pod \"9c57d49a-895e-4077-8c68-4b7637713d87\" (UID: \"9c57d49a-895e-4077-8c68-4b7637713d87\") "
Oct 30 03:32:16.278158 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.277957    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\" (UniqueName: \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\") pod \"e32f8695-0896-40d7-9893-ac55d5d161b0\" (UID: \"e32f8695-0896-40d7-9893-ac55d5d161b0\") "
Oct 30 03:32:16.278244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.278147    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume" pod "e32f8695-0896-40d7-9893-ac55d5d161b0" (UID: "e32f8695-0896-40d7-9893-ac55d5d161b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:16.278320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.278286    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume" pod "9c57d49a-895e-4077-8c68-4b7637713d87" (UID: "9c57d49a-895e-4077-8c68-4b7637713d87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:16.278384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.278322    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f8695-0896-40d7-9893-ac55d5d161b0-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:16.280662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.280632    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6" pod "9c57d49a-895e-4077-8c68-4b7637713d87" (UID: "9c57d49a-895e-4077-8c68-4b7637713d87"). InnerVolumeSpecName "kube-api-access-f4rs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:16.280805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.280708    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49" pod "e32f8695-0896-40d7-9893-ac55d5d161b0" (UID: "e32f8695-0896-40d7-9893-ac55d5d161b0"). InnerVolumeSpecName "kube-api-access-28c49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:16.379528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.379432    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rs6\" (UniqueName: \"kubernetes.io/projected/9c57d49a-895e-4077-8c68-4b7637713d87-kube-api-access-f4rs6\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:16.379528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.379464    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28c49\" (UniqueName: \"kubernetes.io/projected/e32f8695-0896-40d7-9893-ac55d5d161b0-kube-api-access-28c49\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:16.379528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.379476    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c57d49a-895e-4077-8c68-4b7637713d87-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:16.885309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.885270    4214 generic.go:356] "Generic (PLEG): container finished" podID="9c57d49a-895e-4077-8c68-4b7637713d87" containerID="561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b" exitCode=0
Oct 30 03:32:16.885516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.885326    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"
Oct 30 03:32:16.885516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.885363    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" event={"ID":"9c57d49a-895e-4077-8c68-4b7637713d87","Type":"ContainerDied","Data":"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"}
Oct 30 03:32:16.885516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.885425    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9" event={"ID":"9c57d49a-895e-4077-8c68-4b7637713d87","Type":"ContainerDied","Data":"7504f166b421c8670d1c5a688347a41aea2529c144a6a5fbe2171491ac935b0f"}
Oct 30 03:32:16.885516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.885457    4214 scope.go:122] "RemoveContainer" containerID="561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"
Oct 30 03:32:16.886865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.886840    4214 generic.go:356] "Generic (PLEG): container finished" podID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerID="5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3" exitCode=0
Oct 30 03:32:16.887061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.886888    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"
Oct 30 03:32:16.887061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.886926    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" event={"ID":"e32f8695-0896-40d7-9893-ac55d5d161b0","Type":"ContainerDied","Data":"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"}
Oct 30 03:32:16.887061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.886991    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf" event={"ID":"e32f8695-0896-40d7-9893-ac55d5d161b0","Type":"ContainerDied","Data":"d0c5f36e37107f69b215ffae57803da0e2174165def7f3479a725defd12b8227"}
Oct 30 03:32:16.895387 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.895369    4214 scope.go:122] "RemoveContainer" containerID="561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"
Oct 30 03:32:16.895807 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:16.895784    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b\": container with ID starting with 561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b not found: ID does not exist" containerID="561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"
Oct 30 03:32:16.895863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.895816    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b"} err="failed to get container status \"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b\": rpc error: code = NotFound desc = could not find container \"561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b\": container with ID starting with 561d3bc6913f2b1a4cd574179e8cca514ed514402f6f4da74dac6e965936db4b not found: ID does not exist"
Oct 30 03:32:16.895863 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.895832    4214 scope.go:122] "RemoveContainer" containerID="5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"
Oct 30 03:32:16.904845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.904829    4214 scope.go:122] "RemoveContainer" containerID="5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"
Oct 30 03:32:16.905141 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:16.905123    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3\": container with ID starting with 5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3 not found: ID does not exist" containerID="5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"
Oct 30 03:32:16.905198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.905150    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3"} err="failed to get container status \"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3\": rpc error: code = NotFound desc = could not find container \"5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3\": container with ID starting with 5e52e193b5355935413a57aa665fd3caf3e1f61aa976c7cb129dd32135a84bf3 not found: ID does not exist"
Oct 30 03:32:16.939548 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.939517    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:32:16.952443 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.952417    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d823d1-5c9656fc48-qj5cf"]
Oct 30 03:32:16.990249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.990200    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:32:16.993042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:16.993014    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d823d1-57f4d97b55-bwhz9"]
Oct 30 03:32:17.064732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:17.064699    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" path="/var/lib/kubelet/pods/9c57d49a-895e-4077-8c68-4b7637713d87/volumes"
Oct 30 03:32:17.065090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:17.065077    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" path="/var/lib/kubelet/pods/e32f8695-0896-40d7-9893-ac55d5d161b0/volumes"
Oct 30 03:32:17.166208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:17.166101    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing" containerID="cri-o://8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad" gracePeriod=25
Oct 30 03:32:17.192857 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:17.192810    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing" containerID="cri-o://29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118" gracePeriod=25
Oct 30 03:32:20.649271 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:20.649219    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:32:20.649758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:20.649531    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" containerID="cri-o://9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621" gracePeriod=30
Oct 30 03:32:21.322449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322413    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"]
Oct 30 03:32:21.322838 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322826    4214 cpu_manager.go:414] "containerMap: removing container" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322839    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322852    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322857    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322864    4214 cpu_manager.go:414] "containerMap: removing container" podUID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" containerName="maas-api"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322872    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" containerName="maas-api"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322883    4214 cpu_manager.go:414] "containerMap: removing container" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:32:21.322886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322888    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015c99fd-736c-4585-a94e-c20ecac4f1f4" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322901    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322906    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322913    4214 cpu_manager.go:414] "containerMap: removing container" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322918    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322925    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322929    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322936    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322940    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322945    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322949    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322958    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322963    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322974    4214 cpu_manager.go:414] "containerMap: removing container" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322978    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322984    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322988    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.322996    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6a3e0c57-eed0-4005-99f5-3c03171418d2" containerName="main"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323000    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3e0c57-eed0-4005-99f5-3c03171418d2" containerName="main"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323006    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323011    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323040    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1a152e51-e5bf-426d-9e44-4b4c862971e1" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323055    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a152e51-e5bf-426d-9e44-4b4c862971e1" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323064    4214 cpu_manager.go:414] "containerMap: removing container" podUID="ede07f81-f152-4233-af36-5af83c59dec3" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323069    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede07f81-f152-4233-af36-5af83c59dec3" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323075    4214 cpu_manager.go:414] "containerMap: removing container" podUID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" containerName="maas-api"
Oct 30 03:32:21.323121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323080    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" containerName="maas-api"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323139    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323148    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32f8695-0896-40d7-9893-ac55d5d161b0" containerName="payload-processing"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323155    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323161    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3e0c57-eed0-4005-99f5-3c03171418d2" containerName="main"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323167    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323174    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323180    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a152e51-e5bf-426d-9e44-4b4c862971e1" containerName="maas-api"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323185    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede07f81-f152-4233-af36-5af83c59dec3" containerName="maas-api"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323190    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="559ed535-5e2e-4e43-9a1c-0b38dd3f2843" containerName="maas-api"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323195    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="61881be2-60e7-480b-8ab8-40544091d90e" containerName="payload-pre-processing"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323200    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f0cdb8-ec94-4a1f-a68e-80ee33db0a50" containerName="maas-api"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323206    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fb0d90-45b4-454f-95f3-ae7c84e2448f" containerName="payload-processing"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323211    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c57d49a-895e-4077-8c68-4b7637713d87" containerName="payload-pre-processing"
Oct 30 03:32:21.324018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.323215    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40f86a2-bcfa-41f4-a00c-433ebea44c71" containerName="maas-api"
Oct 30 03:32:21.327932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.327909    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.353086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.353054    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"]
Oct 30 03:32:21.399475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.399450    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:32:21.399475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.399459    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:32:21.428953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.428920    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9d6w\" (UniqueName: \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.429142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.428982    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.429142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.429085    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.529999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.529965    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9d6w\" (UniqueName: \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.530186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.530030    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.530186 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.530066    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.530859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.530833    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.532757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.532736    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.550073 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.550044    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9d6w\" (UniqueName: \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\") pod \"maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.642084 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.642012    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:21.942483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:21.942444    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"]
Oct 30 03:32:21.943817 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:32:21.943786    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e7cc97_6964_449a_bfaf_cf4aae2176c3.slice/crio-fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba WatchSource:0}: Error finding container fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba: Status 404 returned error can't find the container with id fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba
Oct 30 03:32:22.928561 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:22.928524    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr" event={"ID":"d7e7cc97-6964-449a-bfaf-cf4aae2176c3","Type":"ContainerStarted","Data":"7c03e3dcfc300dd12e3eb1bfb29ab4085af3dcb18677cf0fa3a72e7e02003aa7"}
Oct 30 03:32:22.928740 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:22.928578    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr" event={"ID":"d7e7cc97-6964-449a-bfaf-cf4aae2176c3","Type":"ContainerStarted","Data":"fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba"}
Oct 30 03:32:22.971525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:22.971478    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr" podStartSLOduration=1.971463948 podStartE2EDuration="1.971463948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:32:22.970500928 +0000 UTC m=+3082.044671720" watchObservedRunningTime="2026-10-30 03:32:22.971463948 +0000 UTC m=+3082.045634741"
Oct 30 03:32:23.380153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.380114    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:23.380372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.380119    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:23.936289 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.936224    4214 generic.go:356] "Generic (PLEG): container finished" podID="d7e7cc97-6964-449a-bfaf-cf4aae2176c3" containerID="7c03e3dcfc300dd12e3eb1bfb29ab4085af3dcb18677cf0fa3a72e7e02003aa7" exitCode=0
Oct 30 03:32:23.936493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.936304    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr" event={"ID":"d7e7cc97-6964-449a-bfaf-cf4aae2176c3","Type":"ContainerDied","Data":"7c03e3dcfc300dd12e3eb1bfb29ab4085af3dcb18677cf0fa3a72e7e02003aa7"}
Oct 30 03:32:23.962801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.962759    4214 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:32:23.962973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:23.962819    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:32:25.083375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.083350    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:25.163719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.163681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\") pod \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") "
Oct 30 03:32:25.163926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.163780    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\" (UniqueName: \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\") pod \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") "
Oct 30 03:32:25.163926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.163825    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\") pod \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\" (UID: \"d7e7cc97-6964-449a-bfaf-cf4aae2176c3\") "
Oct 30 03:32:25.164206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.164182    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca" pod "d7e7cc97-6964-449a-bfaf-cf4aae2176c3" (UID: "d7e7cc97-6964-449a-bfaf-cf4aae2176c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:25.165927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.165905    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca" pod "d7e7cc97-6964-449a-bfaf-cf4aae2176c3" (UID: "d7e7cc97-6964-449a-bfaf-cf4aae2176c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:25.166015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.165984    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w" pod "d7e7cc97-6964-449a-bfaf-cf4aae2176c3" (UID: "d7e7cc97-6964-449a-bfaf-cf4aae2176c3"). InnerVolumeSpecName "kube-api-access-c9d6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:25.264710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.264621    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.264710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.264656    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.264710 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.264666    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9d6w\" (UniqueName: \"kubernetes.io/projected/d7e7cc97-6964-449a-bfaf-cf4aae2176c3-kube-api-access-c9d6w\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.896892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.896864    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:32:25.950378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.950284    4214 generic.go:356] "Generic (PLEG): container finished" podID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerID="9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621" exitCode=0
Oct 30 03:32:25.950378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.950343    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"
Oct 30 03:32:25.950378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.950358    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerDied","Data":"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"}
Oct 30 03:32:25.950641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.950417    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt" event={"ID":"4f8da9c6-da55-45d6-9f70-f71f00866b63","Type":"ContainerDied","Data":"91f8ad99ead6559180db7c17b1b48c2be1e5da51bc13411be2777fc1d8ae24f2"}
Oct 30 03:32:25.950641 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.950444    4214 scope.go:122] "RemoveContainer" containerID="9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"
Oct 30 03:32:25.951761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.951744    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr" event={"ID":"d7e7cc97-6964-449a-bfaf-cf4aae2176c3","Type":"ContainerDied","Data":"fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba"}
Oct 30 03:32:25.951854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.951777    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa567523a9e976e69b795eca96ca230c6aca7e3390d4cd69610c1d46abc2a2ba"
Oct 30 03:32:25.951854 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.951789    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"
Oct 30 03:32:25.963323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.963306    4214 scope.go:122] "RemoveContainer" containerID="0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"
Oct 30 03:32:25.971661 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971622    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.971802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971681    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.971802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971743    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.971802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971766    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.971802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971785    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.972044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971817    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.972044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971915    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.972044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971925    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:25.972044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971988    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.972044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.971995    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:25.972342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972062    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\") pod \"4f8da9c6-da55-45d6-9f70-f71f00866b63\" (UID: \"4f8da9c6-da55-45d6-9f70-f71f00866b63\") "
Oct 30 03:32:25.972342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972104    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:25.972463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972391    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:25.972463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972408    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:25.972584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972459    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.972584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972482    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.972584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.972496    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:25.974296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.974269    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:25.974669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.974636    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "kube-api-access-xgmpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:25.974773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.974738    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:25.974853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.974835    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo" pod "4f8da9c6-da55-45d6-9f70-f71f00866b63" (UID: "4f8da9c6-da55-45d6-9f70-f71f00866b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:32:25.975858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.975835    4214 scope.go:122] "RemoveContainer" containerID="9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"
Oct 30 03:32:25.976328 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:25.976297    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621\": container with ID starting with 9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621 not found: ID does not exist" containerID="9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"
Oct 30 03:32:25.976423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.976341    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621"} err="failed to get container status \"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621\": rpc error: code = NotFound desc = could not find container \"9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621\": container with ID starting with 9247e6e55fbcacb9ff356c86089503e030d79a03b5db27c41d15e5ffa8bed621 not found: ID does not exist"
Oct 30 03:32:25.976423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.976369    4214 scope.go:122] "RemoveContainer" containerID="0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"
Oct 30 03:32:25.976700 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:25.976683    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3\": container with ID starting with 0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3 not found: ID does not exist" containerID="0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"
Oct 30 03:32:25.976774 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:25.976707    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3"} err="failed to get container status \"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3\": rpc error: code = NotFound desc = could not find container \"0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3\": container with ID starting with 0d6c426d004a87a0df618f452ad0b9715753d17aa9af2c60c4f87af3e4f06bc3 not found: ID does not exist"
Oct 30 03:32:26.073070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073036    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8da9c6-da55-45d6-9f70-f71f00866b63-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.073070 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073073    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmpr\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-kube-api-access-xgmpr\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.073371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073088    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.073371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073104    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.073371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073118    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8da9c6-da55-45d6-9f70-f71f00866b63-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.073371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.073132    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8da9c6-da55-45d6-9f70-f71f00866b63-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.301133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.301099    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:32:26.313907 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:26.313872    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-hl6pt"]
Oct 30 03:32:27.065151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:27.065118    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" path="/var/lib/kubelet/pods/4f8da9c6-da55-45d6-9f70-f71f00866b63/volumes"
Oct 30 03:32:31.777202 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:31.777164    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 03:32:33.380011 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.379976    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:33.380434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.380077    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:33.506435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506394    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"]
Oct 30 03:32:33.506826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506814    4214 cpu_manager.go:414] "containerMap: removing container" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.506873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506828    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.506873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506853    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d7e7cc97-6964-449a-bfaf-cf4aae2176c3" containerName="revoke-keys"
Oct 30 03:32:33.506873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506863    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e7cc97-6964-449a-bfaf-cf4aae2176c3" containerName="revoke-keys"
Oct 30 03:32:33.506994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506886    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:33.506994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506893    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:33.506994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506904    4214 cpu_manager.go:414] "containerMap: removing container" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.506994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.506911    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507008    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfa4bd8-5357-452a-80ea-0f694fe8e6fc" containerName="istio-proxy"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507023    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e7cc97-6964-449a-bfaf-cf4aae2176c3" containerName="revoke-keys"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507036    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c415a63-7b04-4ee7-be1a-817c1a465fa3" containerName="cleanup"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507056    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507070    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f592998c-cb32-4292-a4ac-86eca80e305e" containerName="istio-proxy"
Oct 30 03:32:33.507159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.507079    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8da9c6-da55-45d6-9f70-f71f00866b63" containerName="istio-proxy"
Oct 30 03:32:33.510180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.510161    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.539369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.539338    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:32:33.539369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.539364    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:32:33.541738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.541710    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"]
Oct 30 03:32:33.550974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.550942    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.551175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.551154    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.551306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.551213    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67tj\" (UniqueName: \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.653102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.653002    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.653102 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.653069    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67tj\" (UniqueName: \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.653379 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.653139    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.653981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.653951    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.655614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.655588    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.672973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.672950    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67tj\" (UniqueName: \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\") pod \"maas-api-revoke-sub-1cbcc9b7-xgvtb\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:33.823893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:33.823860    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:34.122434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:34.122405    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"]
Oct 30 03:32:34.124328 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:32:34.124298    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff0bb34_f3e0_4a2a_a269_0ca716ee1589.slice/crio-acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50 WatchSource:0}: Error finding container acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50: Status 404 returned error can't find the container with id acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50
Oct 30 03:32:35.006458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:35.006427    4214 generic.go:356] "Generic (PLEG): container finished" podID="2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" containerID="1c200b9801e602eaf75db445955afb23dce4304775a67994ec39b5b9b330cf5b" exitCode=0
Oct 30 03:32:35.006859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:35.006472    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb" event={"ID":"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589","Type":"ContainerDied","Data":"1c200b9801e602eaf75db445955afb23dce4304775a67994ec39b5b9b330cf5b"}
Oct 30 03:32:35.006859 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:35.006521    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb" event={"ID":"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589","Type":"ContainerStarted","Data":"acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50"}
Oct 30 03:32:36.167118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.167093    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:36.277625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.277548    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\" (UniqueName: \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\") pod \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") "
Oct 30 03:32:36.277777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.277695    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\") pod \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") "
Oct 30 03:32:36.277777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.277717    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\") pod \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\" (UID: \"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589\") "
Oct 30 03:32:36.278137 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.278110    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca" pod "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" (UID: "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:36.279666 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.279641    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca" pod "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" (UID: "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:36.279768 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.279705    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj" pod "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" (UID: "2ff0bb34-f3e0-4a2a-a269-0ca716ee1589"). InnerVolumeSpecName "kube-api-access-b67tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:36.379275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.379217    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b67tj\" (UniqueName: \"kubernetes.io/projected/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-kube-api-access-b67tj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:36.379275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.379271    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:36.379275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.379282    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:36.778473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:36.778430    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing" containerID="cri-o://94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730" gracePeriod=25
Oct 30 03:32:37.018937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.018909    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"
Oct 30 03:32:37.018937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.018928    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb" event={"ID":"2ff0bb34-f3e0-4a2a-a269-0ca716ee1589","Type":"ContainerDied","Data":"acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50"}
Oct 30 03:32:37.019164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.018974    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf4a770c613eef0816b028c1d9e73d68bdaea30c65dc8d661dbe18e9df60d50"
Oct 30 03:32:37.438394 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.438367    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:32:37.464967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.464941    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:32:37.489722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.489693    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\" (UniqueName: \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\") pod \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") "
Oct 30 03:32:37.489872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.489858    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\") pod \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\" (UID: \"7c10691e-c6f6-4ccc-b5e2-4155217cca9a\") "
Oct 30 03:32:37.490199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.490175    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume" pod "7c10691e-c6f6-4ccc-b5e2-4155217cca9a" (UID: "7c10691e-c6f6-4ccc-b5e2-4155217cca9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:37.491963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.491937    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58" pod "7c10691e-c6f6-4ccc-b5e2-4155217cca9a" (UID: "7c10691e-c6f6-4ccc-b5e2-4155217cca9a"). InnerVolumeSpecName "kube-api-access-zlq58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:37.591295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.591174    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\") pod \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") "
Oct 30 03:32:37.591295 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.591217    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\" (UniqueName: \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\") pod \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\" (UID: \"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c\") "
Oct 30 03:32:37.591555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.591485    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:37.591555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.591496    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlq58\" (UniqueName: \"kubernetes.io/projected/7c10691e-c6f6-4ccc-b5e2-4155217cca9a-kube-api-access-zlq58\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:37.591682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.591656    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume" pod "d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" (UID: "d5d0d633-6da9-4738-ba5f-f841a8aa9f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:37.593393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.593371    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd" pod "d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" (UID: "d5d0d633-6da9-4738-ba5f-f841a8aa9f2c"). InnerVolumeSpecName "kube-api-access-zsjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:37.692674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.692637    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:37.692674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:37.692668    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjfd\" (UniqueName: \"kubernetes.io/projected/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c-kube-api-access-zsjfd\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:38.024962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.024877    4214 generic.go:356] "Generic (PLEG): container finished" podID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerID="29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118" exitCode=0
Oct 30 03:32:38.024962 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.024917    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"
Oct 30 03:32:38.025217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.024958    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" event={"ID":"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c","Type":"ContainerDied","Data":"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"}
Oct 30 03:32:38.025217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.025017    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn" event={"ID":"d5d0d633-6da9-4738-ba5f-f841a8aa9f2c","Type":"ContainerDied","Data":"6d9aa88f16be69cadfbd64a34aeede0425249ddbb6cb1ecc63982a95f65de38f"}
Oct 30 03:32:38.025217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.025046    4214 scope.go:122] "RemoveContainer" containerID="29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"
Oct 30 03:32:38.026458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.026437    4214 generic.go:356] "Generic (PLEG): container finished" podID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerID="8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad" exitCode=0
Oct 30 03:32:38.026588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.026473    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"
Oct 30 03:32:38.026588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.026523    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" event={"ID":"7c10691e-c6f6-4ccc-b5e2-4155217cca9a","Type":"ContainerDied","Data":"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"}
Oct 30 03:32:38.026588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.026566    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857" event={"ID":"7c10691e-c6f6-4ccc-b5e2-4155217cca9a","Type":"ContainerDied","Data":"02b6e4b51538bd6edb957a6e7f9c6f91829186b752f65a11e0764cfdc74555f5"}
Oct 30 03:32:38.035723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.035704    4214 scope.go:122] "RemoveContainer" containerID="29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"
Oct 30 03:32:38.036005 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:38.035978    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118\": container with ID starting with 29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118 not found: ID does not exist" containerID="29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"
Oct 30 03:32:38.036081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.036013    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118"} err="failed to get container status \"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118\": rpc error: code = NotFound desc = could not find container \"29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118\": container with ID starting with 29552a4eaba6af8201d721a4ee47b29afeb04afac1231e97d5f85995788f0118 not found: ID does not exist"
Oct 30 03:32:38.036081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.036029    4214 scope.go:122] "RemoveContainer" containerID="8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"
Oct 30 03:32:38.047625 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.047607    4214 scope.go:122] "RemoveContainer" containerID="8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"
Oct 30 03:32:38.047891 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:38.047873    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad\": container with ID starting with 8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad not found: ID does not exist" containerID="8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"
Oct 30 03:32:38.047942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.047899    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad"} err="failed to get container status \"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad\": rpc error: code = NotFound desc = could not find container \"8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad\": container with ID starting with 8736e36055ac70bd6ef7f9004714aae13e1be8acfbc212e729edbfa8f68b0fad not found: ID does not exist"
Oct 30 03:32:38.077360 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.077325    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:32:38.090293 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.090254    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-874ab5-79d6bdcf7b-q4857"]
Oct 30 03:32:38.115648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.115612    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:32:38.118372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:38.118349    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-874ab5-fbc9f8659-mxzrn"]
Oct 30 03:32:39.065342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:39.065309    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" path="/var/lib/kubelet/pods/7c10691e-c6f6-4ccc-b5e2-4155217cca9a/volumes"
Oct 30 03:32:39.065734 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:39.065697    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" path="/var/lib/kubelet/pods/d5d0d633-6da9-4738-ba5f-f841a8aa9f2c/volumes"
Oct 30 03:32:39.163483 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:39.163444    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:32:40.578939 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:40.578902    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:49.216258 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:49.216161    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"]
Oct 30 03:32:49.230728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:49.230687    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-aeff1d-6hqqr"]
Oct 30 03:32:50.579091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:50.579043    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:51.064785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:51.064747    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e7cc97-6964-449a-bfaf-cf4aae2176c3" path="/var/lib/kubelet/pods/d7e7cc97-6964-449a-bfaf-cf4aae2176c3/volumes"
Oct 30 03:32:57.044572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.044550    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 03:32:57.134079 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.134044    4214 generic.go:356] "Generic (PLEG): container finished" podID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerID="94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730" exitCode=0
Oct 30 03:32:57.134292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.134097    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"
Oct 30 03:32:57.134292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.134133    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" event={"ID":"1df033e2-92e3-40b1-9552-ae4fe56cd446","Type":"ContainerDied","Data":"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"}
Oct 30 03:32:57.134292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.134194    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z" event={"ID":"1df033e2-92e3-40b1-9552-ae4fe56cd446","Type":"ContainerDied","Data":"a446bacc5f54591204b6e915cfd05188d33e863da95333c6846f37980ead4a28"}
Oct 30 03:32:57.134292 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.134222    4214 scope.go:122] "RemoveContainer" containerID="94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"
Oct 30 03:32:57.143593 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.143572    4214 scope.go:122] "RemoveContainer" containerID="94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"
Oct 30 03:32:57.143900 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:57.143879    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730\": container with ID starting with 94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730 not found: ID does not exist" containerID="94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"
Oct 30 03:32:57.143974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.143916    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730"} err="failed to get container status \"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730\": rpc error: code = NotFound desc = could not find container \"94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730\": container with ID starting with 94955047ff21b8382024b039716c087078d8bd4c36f8fa4dabc3b0ed150e9730 not found: ID does not exist"
Oct 30 03:32:57.185140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.185096    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\" (UniqueName: \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\") pod \"1df033e2-92e3-40b1-9552-ae4fe56cd446\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") "
Oct 30 03:32:57.185367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.185164    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\") pod \"1df033e2-92e3-40b1-9552-ae4fe56cd446\" (UID: \"1df033e2-92e3-40b1-9552-ae4fe56cd446\") "
Oct 30 03:32:57.185612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.185573    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume" pod "1df033e2-92e3-40b1-9552-ae4fe56cd446" (UID: "1df033e2-92e3-40b1-9552-ae4fe56cd446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:57.187317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.187287    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck" pod "1df033e2-92e3-40b1-9552-ae4fe56cd446" (UID: "1df033e2-92e3-40b1-9552-ae4fe56cd446"). InnerVolumeSpecName "kube-api-access-kjbck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:57.286921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.286879    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbck\" (UniqueName: \"kubernetes.io/projected/1df033e2-92e3-40b1-9552-ae4fe56cd446-kube-api-access-kjbck\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:57.286921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.286914    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df033e2-92e3-40b1-9552-ae4fe56cd446-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:57.480044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.480005    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 03:32:57.492816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:57.492785    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-aeff1d-5f7d4bc7d7-qs76z"]
Oct 30 03:32:58.799732 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:58.799677    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 03:32:58.800272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:58.799931    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6658f68bfc-xts57" podUID="8f1b3588-f564-430d-b012-411cd460d36d" containerName="manager" containerID="cri-o://fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69" gracePeriod=30
Oct 30 03:32:59.037160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.037133    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 03:32:59.065522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.065437    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" path="/var/lib/kubelet/pods/1df033e2-92e3-40b1-9552-ae4fe56cd446/volumes"
Oct 30 03:32:59.146963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.146919    4214 generic.go:356] "Generic (PLEG): container finished" podID="8f1b3588-f564-430d-b012-411cd460d36d" containerID="fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69" exitCode=0
Oct 30 03:32:59.147162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.146978    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-xts57" event={"ID":"8f1b3588-f564-430d-b012-411cd460d36d","Type":"ContainerDied","Data":"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"}
Oct 30 03:32:59.147162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.147039    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-xts57" event={"ID":"8f1b3588-f564-430d-b012-411cd460d36d","Type":"ContainerDied","Data":"4d77cecf41308188eb86350356de1e82bb9b85abf2eedd859df78b43eacbdaad"}
Oct 30 03:32:59.147162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.147042    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-xts57"
Oct 30 03:32:59.147162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.147116    4214 scope.go:122] "RemoveContainer" containerID="fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"
Oct 30 03:32:59.158862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.158839    4214 scope.go:122] "RemoveContainer" containerID="fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"
Oct 30 03:32:59.159164 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:32:59.159140    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69\": container with ID starting with fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69 not found: ID does not exist" containerID="fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"
Oct 30 03:32:59.159245 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.159178    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69"} err="failed to get container status \"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69\": rpc error: code = NotFound desc = could not find container \"fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69\": container with ID starting with fedee540da9f9340d718d2434cd4f0fff060dfafccd2a5524c578668d66c7e69 not found: ID does not exist"
Oct 30 03:32:59.202392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.202360    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\") pod \"8f1b3588-f564-430d-b012-411cd460d36d\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") "
Oct 30 03:32:59.202550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.202420    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\") pod \"8f1b3588-f564-430d-b012-411cd460d36d\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") "
Oct 30 03:32:59.202550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.202454    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\" (UniqueName: \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\") pod \"8f1b3588-f564-430d-b012-411cd460d36d\" (UID: \"8f1b3588-f564-430d-b012-411cd460d36d\") "
Oct 30 03:32:59.204456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.204422    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg" pod "8f1b3588-f564-430d-b012-411cd460d36d" (UID: "8f1b3588-f564-430d-b012-411cd460d36d"). InnerVolumeSpecName "kube-api-access-779dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:59.204581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.204459    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert" pod "8f1b3588-f564-430d-b012-411cd460d36d" (UID: "8f1b3588-f564-430d-b012-411cd460d36d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:59.204581 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.204497    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs" pod "8f1b3588-f564-430d-b012-411cd460d36d" (UID: "8f1b3588-f564-430d-b012-411cd460d36d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:59.304265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.304213    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:59.304265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.304261    4214 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f1b3588-f564-430d-b012-411cd460d36d-webhook-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:59.304265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.304271    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779dg\" (UniqueName: \"kubernetes.io/projected/8f1b3588-f564-430d-b012-411cd460d36d-kube-api-access-779dg\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:32:59.493574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.493541    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 03:32:59.506277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:32:59.506251    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-xts57"]
Oct 30 03:33:00.192877 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.192845    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:00.193287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193266    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" containerName="revoke-keys"
Oct 30 03:33:00.193287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193283    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" containerName="revoke-keys"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193297    4214 cpu_manager.go:414] "containerMap: removing container" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193306    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193318    4214 cpu_manager.go:414] "containerMap: removing container" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193323    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193340    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193346    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193356    4214 cpu_manager.go:414] "containerMap: removing container" podUID="8f1b3588-f564-430d-b012-411cd460d36d" containerName="manager"
Oct 30 03:33:00.193372 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193361    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1b3588-f564-430d-b012-411cd460d36d" containerName="manager"
Oct 30 03:33:00.193623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193440    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c10691e-c6f6-4ccc-b5e2-4155217cca9a" containerName="payload-processing"
Oct 30 03:33:00.193623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193453    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1b3588-f564-430d-b012-411cd460d36d" containerName="manager"
Oct 30 03:33:00.193623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193466    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" containerName="revoke-keys"
Oct 30 03:33:00.193623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193471    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df033e2-92e3-40b1-9552-ae4fe56cd446" containerName="payload-pre-processing"
Oct 30 03:33:00.193623 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.193478    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d0d633-6da9-4738-ba5f-f841a8aa9f2c" containerName="payload-pre-processing"
Oct 30 03:33:00.197991 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.197972    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.217677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.217652    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:00.228207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.228182    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 03:33:00.228369 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.228223    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 03:33:00.230796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.230770    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 03:33:00.230900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.230848    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7hglq\""
Oct 30 03:33:00.313978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.313932    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.314153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.313990    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.314153 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.314089    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2z75\" (UniqueName: \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.415566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.415531    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.415769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.415576    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.415769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.415614    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2z75\" (UniqueName: \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.423765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.423713    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.424944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.424141    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.436397 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.436369    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2z75\" (UniqueName: \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\") pod \"maas-controller-6658f68bfc-vq9nc\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") " pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.511473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.511379    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:00.817841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.817818    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:00.824619 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:33:00.824592    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a22f52_3123_47c5_ace2_c0a722642f7f.slice/crio-dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1 WatchSource:0}: Error finding container dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1: Status 404 returned error can't find the container with id dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1
Oct 30 03:33:00.826370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:00.826349    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:33:01.065736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:01.065698    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1b3588-f564-430d-b012-411cd460d36d" path="/var/lib/kubelet/pods/8f1b3588-f564-430d-b012-411cd460d36d/volumes"
Oct 30 03:33:01.161138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:01.161055    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" event={"ID":"01a22f52-3123-47c5-ace2-c0a722642f7f","Type":"ContainerStarted","Data":"dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1"}
Oct 30 03:33:01.421903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:01.421848    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 03:33:02.169815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:02.169775    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" event={"ID":"01a22f52-3123-47c5-ace2-c0a722642f7f","Type":"ContainerStarted","Data":"1cb308fa64b589de50db0e1a9e346c4b3d6ee7e5645b6836d4d096cbe1460f97"}
Oct 30 03:33:02.170049 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:02.169856    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:02.211604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:02.211542    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" podStartSLOduration=1.631402695 podStartE2EDuration="2.211524414s" totalImagesPullingTime="580.121719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:02.210715383 +0000 UTC m=+3121.284886222" watchObservedRunningTime="2026-10-30 03:33:02.211524414 +0000 UTC m=+3121.285695207"
Oct 30 03:33:09.163157 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:33:09.163117    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:33:13.178873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:13.178843    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:20.400160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.400121    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"]
Oct 30 03:33:20.405170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.405150    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.423999 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.423971    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"]
Oct 30 03:33:20.434220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.434198    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:33:20.434396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.434346    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:33:20.504743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.504711    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.504932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.504769    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.504932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.504890    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqgb\" (UniqueName: \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.605645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.605610    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.605860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.605672    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.605860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.605717    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqgb\" (UniqueName: \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.606412 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.606388    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.608128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.608094    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.626081 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.626054    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqgb\" (UniqueName: \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\") pod \"maas-api-revoke-sub-85237e45-mgps4\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:20.718348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:20.718264    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:21.054363 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:21.054328    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"]
Oct 30 03:33:21.070612 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:33:21.070564    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de56eba_3ad6_4a03_9d1f_847450af80a8.slice/crio-b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb WatchSource:0}: Error finding container b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb: Status 404 returned error can't find the container with id b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb
Oct 30 03:33:21.275016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:21.274973    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4" event={"ID":"6de56eba-3ad6-4a03-9d1f-847450af80a8","Type":"ContainerStarted","Data":"b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb"}
Oct 30 03:33:22.282328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:22.282199    4214 generic.go:356] "Generic (PLEG): container finished" podID="6de56eba-3ad6-4a03-9d1f-847450af80a8" containerID="c69094fbc3687041d00afa9d5bb37214de507e401e15954897d34cc8b91100bd" exitCode=0
Oct 30 03:33:22.282328 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:22.282279    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4" event={"ID":"6de56eba-3ad6-4a03-9d1f-847450af80a8","Type":"ContainerDied","Data":"c69094fbc3687041d00afa9d5bb37214de507e401e15954897d34cc8b91100bd"}
Oct 30 03:33:23.420060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.420033    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:23.531405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.531369    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\") pod \"6de56eba-3ad6-4a03-9d1f-847450af80a8\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") "
Oct 30 03:33:23.531608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.531417    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\") pod \"6de56eba-3ad6-4a03-9d1f-847450af80a8\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") "
Oct 30 03:33:23.531608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.531554    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\" (UniqueName: \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\") pod \"6de56eba-3ad6-4a03-9d1f-847450af80a8\" (UID: \"6de56eba-3ad6-4a03-9d1f-847450af80a8\") "
Oct 30 03:33:23.531765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.531733    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca" pod "6de56eba-3ad6-4a03-9d1f-847450af80a8" (UID: "6de56eba-3ad6-4a03-9d1f-847450af80a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:33:23.533642 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.533616    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca" pod "6de56eba-3ad6-4a03-9d1f-847450af80a8" (UID: "6de56eba-3ad6-4a03-9d1f-847450af80a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:33:23.533737 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.533638    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb" pod "6de56eba-3ad6-4a03-9d1f-847450af80a8" (UID: "6de56eba-3ad6-4a03-9d1f-847450af80a8"). InnerVolumeSpecName "kube-api-access-zzqgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:23.632735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.632696    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqgb\" (UniqueName: \"kubernetes.io/projected/6de56eba-3ad6-4a03-9d1f-847450af80a8-kube-api-access-zzqgb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:23.632735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.632729    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:23.632944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:23.632744    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de56eba-3ad6-4a03-9d1f-847450af80a8-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:24.292729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:24.292690    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"
Oct 30 03:33:24.292917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:24.292725    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4" event={"ID":"6de56eba-3ad6-4a03-9d1f-847450af80a8","Type":"ContainerDied","Data":"b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb"}
Oct 30 03:33:24.292917 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:24.292774    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0013f05fc77db48bd845e595dc00144f36940894c3a12e0fd453f754fa8c3bb"
Oct 30 03:33:32.082978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.082935    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:32.083608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.083192    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" podUID="01a22f52-3123-47c5-ace2-c0a722642f7f" containerName="manager" containerID="cri-o://1cb308fa64b589de50db0e1a9e346c4b3d6ee7e5645b6836d4d096cbe1460f97" gracePeriod=30
Oct 30 03:33:32.336127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.336058    4214 generic.go:356] "Generic (PLEG): container finished" podID="01a22f52-3123-47c5-ace2-c0a722642f7f" containerID="1cb308fa64b589de50db0e1a9e346c4b3d6ee7e5645b6836d4d096cbe1460f97" exitCode=0
Oct 30 03:33:32.336262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.336131    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" event={"ID":"01a22f52-3123-47c5-ace2-c0a722642f7f","Type":"ContainerDied","Data":"1cb308fa64b589de50db0e1a9e346c4b3d6ee7e5645b6836d4d096cbe1460f97"}
Oct 30 03:33:32.336262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.336186    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-vq9nc" event={"ID":"01a22f52-3123-47c5-ace2-c0a722642f7f","Type":"ContainerDied","Data":"dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1"}
Oct 30 03:33:32.336262 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.336205    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9f43294a5a0f1a6cfbe953c165fa20f0fed49936027c61a528dfaf91d2e3c1"
Oct 30 03:33:32.341135 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.341113    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:32.414898 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.414853    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\" (UniqueName: \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\") pod \"01a22f52-3123-47c5-ace2-c0a722642f7f\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") "
Oct 30 03:33:32.415133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.414937    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\") pod \"01a22f52-3123-47c5-ace2-c0a722642f7f\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") "
Oct 30 03:33:32.415133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.415010    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\") pod \"01a22f52-3123-47c5-ace2-c0a722642f7f\" (UID: \"01a22f52-3123-47c5-ace2-c0a722642f7f\") "
Oct 30 03:33:32.417242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.417191    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert" pod "01a22f52-3123-47c5-ace2-c0a722642f7f" (UID: "01a22f52-3123-47c5-ace2-c0a722642f7f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:33:32.417550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.417518    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs" pod "01a22f52-3123-47c5-ace2-c0a722642f7f" (UID: "01a22f52-3123-47c5-ace2-c0a722642f7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:33:32.417651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.417528    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75" pod "01a22f52-3123-47c5-ace2-c0a722642f7f" (UID: "01a22f52-3123-47c5-ace2-c0a722642f7f"). InnerVolumeSpecName "kube-api-access-q2z75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:32.516348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.516320    4214 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-webhook-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:32.516521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.516365    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2z75\" (UniqueName: \"kubernetes.io/projected/01a22f52-3123-47c5-ace2-c0a722642f7f-kube-api-access-q2z75\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:32.516521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:32.516375    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a22f52-3123-47c5-ace2-c0a722642f7f-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:33:33.341348 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:33.341263    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-vq9nc"
Oct 30 03:33:33.394522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:33.394488    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:33.407728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:33.407694    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-vq9nc"]
Oct 30 03:33:34.677316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.677280    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6658f68bfc-lz8dz"]
Oct 30 03:33:34.677906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.677864    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6de56eba-3ad6-4a03-9d1f-847450af80a8" containerName="revoke-keys"
Oct 30 03:33:34.677906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.677884    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de56eba-3ad6-4a03-9d1f-847450af80a8" containerName="revoke-keys"
Oct 30 03:33:34.678062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.677910    4214 cpu_manager.go:414] "containerMap: removing container" podUID="01a22f52-3123-47c5-ace2-c0a722642f7f" containerName="manager"
Oct 30 03:33:34.678062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.677919    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a22f52-3123-47c5-ace2-c0a722642f7f" containerName="manager"
Oct 30 03:33:34.678062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.678015    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de56eba-3ad6-4a03-9d1f-847450af80a8" containerName="revoke-keys"
Oct 30 03:33:34.678062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.678031    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a22f52-3123-47c5-ace2-c0a722642f7f" containerName="manager"
Oct 30 03:33:34.680974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.680954    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.696881 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.696859    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 03:33:34.709667 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.709648    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7hglq\""
Oct 30 03:33:34.709820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.709777    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 03:33:34.709820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.709797    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 03:33:34.712353 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.712331    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-lz8dz"]
Oct 30 03:33:34.839321 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.839280    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-webhook-cert\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.839506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.839349    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-metrics-certs\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.839506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.839429    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzpg\" (UniqueName: \"kubernetes.io/projected/467b1e74-a20c-4551-97a1-899a1e1f6ae0-kube-api-access-rkzpg\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.940662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.940572    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzpg\" (UniqueName: \"kubernetes.io/projected/467b1e74-a20c-4551-97a1-899a1e1f6ae0-kube-api-access-rkzpg\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.940662 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.940660    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-webhook-cert\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.940916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.940699    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-metrics-certs\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.943224 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.943200    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-webhook-cert\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.943358 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.943336    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467b1e74-a20c-4551-97a1-899a1e1f6ae0-metrics-certs\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.960477 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.960452    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzpg\" (UniqueName: \"kubernetes.io/projected/467b1e74-a20c-4551-97a1-899a1e1f6ae0-kube-api-access-rkzpg\") pod \"maas-controller-6658f68bfc-lz8dz\" (UID: \"467b1e74-a20c-4551-97a1-899a1e1f6ae0\") " pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:34.993118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:34.993088    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:35.064457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.064422    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a22f52-3123-47c5-ace2-c0a722642f7f" path="/var/lib/kubelet/pods/01a22f52-3123-47c5-ace2-c0a722642f7f/volumes"
Oct 30 03:33:35.193862 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.193778    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"]
Oct 30 03:33:35.198789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.198762    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.230393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.230356    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"]
Oct 30 03:33:35.294085 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.294049    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6658f68bfc-lz8dz"]
Oct 30 03:33:35.295541 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:33:35.295517    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467b1e74_a20c_4551_97a1_899a1e1f6ae0.slice/crio-9c813f731394cd590da1aa4b33fbcb1c730ac206f0d26af4a7a273089cbc795b WatchSource:0}: Error finding container 9c813f731394cd590da1aa4b33fbcb1c730ac206f0d26af4a7a273089cbc795b: Status 404 returned error can't find the container with id 9c813f731394cd590da1aa4b33fbcb1c730ac206f0d26af4a7a273089cbc795b
Oct 30 03:33:35.344088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344055    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583b27ad-575a-4c4c-b3aa-9f512e025a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344088 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344086    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344181    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344338 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c44q\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-kube-api-access-5c44q\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344417 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344344    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344413    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344445    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344505    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.344556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.344534    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.352766 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.352741    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-lz8dz" event={"ID":"467b1e74-a20c-4551-97a1-899a1e1f6ae0","Type":"ContainerStarted","Data":"9c813f731394cd590da1aa4b33fbcb1c730ac206f0d26af4a7a273089cbc795b"}
Oct 30 03:33:35.445781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445698    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c44q\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-kube-api-access-5c44q\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.445781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445754    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.445781 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445779    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445800    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445829    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.445855    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446094    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583b27ad-575a-4c4c-b3aa-9f512e025a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446138    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446166    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446278    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446322    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446449    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446496    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.446746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.446724    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583b27ad-575a-4c4c-b3aa-9f512e025a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.448280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.448261    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.448508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.448491    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.465632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.465602    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.465735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.465724    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c44q\" (UniqueName: \"kubernetes.io/projected/583b27ad-575a-4c4c-b3aa-9f512e025a17-kube-api-access-5c44q\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qv642\" (UID: \"583b27ad-575a-4c4c-b3aa-9f512e025a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.513308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.513279    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:35.826704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.826673    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"]
Oct 30 03:33:35.828387 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:33:35.828361    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583b27ad_575a_4c4c_b3aa_9f512e025a17.slice/crio-1eecc51568f4dfe873f52cc75bc86d0432627fecb80899764d3f09835b1c63fd WatchSource:0}: Error finding container 1eecc51568f4dfe873f52cc75bc86d0432627fecb80899764d3f09835b1c63fd: Status 404 returned error can't find the container with id 1eecc51568f4dfe873f52cc75bc86d0432627fecb80899764d3f09835b1c63fd
Oct 30 03:33:35.830721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.830683    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:35.830806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.830753    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:35.830806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:35.830782    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:33:36.364473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.364428    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6658f68bfc-lz8dz" event={"ID":"467b1e74-a20c-4551-97a1-899a1e1f6ae0","Type":"ContainerStarted","Data":"df1f4bef75e13954ac285993f16dfe1845f3af15ea3de81d0009b450643a6ab0"}
Oct 30 03:33:36.364676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.364497    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:33:36.366133 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.366107    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" event={"ID":"583b27ad-575a-4c4c-b3aa-9f512e025a17","Type":"ContainerStarted","Data":"7b9a2c9bc26b595370f7f74493b155664161eacd0edf77fd6f20a5dd40e9ac58"}
Oct 30 03:33:36.366279 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.366152    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" event={"ID":"583b27ad-575a-4c4c-b3aa-9f512e025a17","Type":"ContainerStarted","Data":"1eecc51568f4dfe873f52cc75bc86d0432627fecb80899764d3f09835b1c63fd"}
Oct 30 03:33:36.433274 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.433199    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6658f68bfc-lz8dz" podStartSLOduration=2.139527707 podStartE2EDuration="2.433184115s" totalImagesPullingTime="293.656408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:36.418462603 +0000 UTC m=+3155.492633396" watchObservedRunningTime="2026-10-30 03:33:36.433184115 +0000 UTC m=+3155.507354907"
Oct 30 03:33:36.463029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.462973    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podStartSLOduration=1.462950064 podStartE2EDuration="1.462950064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:36.460673562 +0000 UTC m=+3155.534844353" watchObservedRunningTime="2026-10-30 03:33:36.462950064 +0000 UTC m=+3155.537120856"
Oct 30 03:33:36.513872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.513833    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:36.515550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.515522    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:36.515694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:36.515591    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:37.514416 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:37.514381    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:37.514899 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:37.514439    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:38.514124 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:38.514088    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:38.514303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:38.514136    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:39.183484 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:33:39.183447    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:33:39.513969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:39.513885    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:39.513969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:39.513936    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:40.514588 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:40.514545    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:40.514993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:40.514620    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:41.513773 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:41.513737    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:41.513968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:41.513792    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:42.513888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:42.513853    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body=
Oct 30 03:33:42.514331 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:42.513910    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642" podUID="583b27ad-575a-4c4c-b3aa-9f512e025a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused"
Oct 30 03:33:44.092420 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:44.092383    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:44.092889 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:44.092792    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:44.093725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:44.093694    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qv642"
Oct 30 03:33:47.374003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:33:47.373968    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6658f68bfc-lz8dz"
Oct 30 03:34:09.170539 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:34:09.170481    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:34:11.426249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.426207    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"]
Oct 30 03:34:11.429317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.429299    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.449820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.449791    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"]
Oct 30 03:34:11.459164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.459137    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:34:11.459324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.459186    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:34:11.620306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.620276    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pg8\" (UniqueName: \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.620504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.620320    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.620504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.620433    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.721685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.721606    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pg8\" (UniqueName: \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.721685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.721654    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.721888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.721704    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.722437 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.722410    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.724046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.724024    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:11.743032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:11.743007    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pg8\" (UniqueName: \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\") pod \"maas-api-revoke-sub-c1e271a9-qm82z\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:12.041267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:12.041166    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:12.341375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:12.341347    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"]
Oct 30 03:34:12.343602 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:34:12.343572    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fc6fe4_3676_42d5_834f_688b6cf2365c.slice/crio-74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d WatchSource:0}: Error finding container 74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d: Status 404 returned error can't find the container with id 74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d
Oct 30 03:34:12.563752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:12.563716    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z" event={"ID":"57fc6fe4-3676-42d5-834f-688b6cf2365c","Type":"ContainerStarted","Data":"74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d"}
Oct 30 03:34:13.570770 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:13.570735    4214 generic.go:356] "Generic (PLEG): container finished" podID="57fc6fe4-3676-42d5-834f-688b6cf2365c" containerID="68ef0b18ebabb1874ee4d87b4cf67941eb3300923325142aeb1e2807d0c830d2" exitCode=0
Oct 30 03:34:13.571177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:13.570816    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z" event={"ID":"57fc6fe4-3676-42d5-834f-688b6cf2365c","Type":"ContainerDied","Data":"68ef0b18ebabb1874ee4d87b4cf67941eb3300923325142aeb1e2807d0c830d2"}
Oct 30 03:34:14.717114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.717086    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:14.853777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.853693    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\") pod \"57fc6fe4-3676-42d5-834f-688b6cf2365c\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") "
Oct 30 03:34:14.853777 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.853752    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\") pod \"57fc6fe4-3676-42d5-834f-688b6cf2365c\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") "
Oct 30 03:34:14.853973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.853859    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\" (UniqueName: \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\") pod \"57fc6fe4-3676-42d5-834f-688b6cf2365c\" (UID: \"57fc6fe4-3676-42d5-834f-688b6cf2365c\") "
Oct 30 03:34:14.854141 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.854111    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca" pod "57fc6fe4-3676-42d5-834f-688b6cf2365c" (UID: "57fc6fe4-3676-42d5-834f-688b6cf2365c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:14.855825 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.855795    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca" pod "57fc6fe4-3676-42d5-834f-688b6cf2365c" (UID: "57fc6fe4-3676-42d5-834f-688b6cf2365c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:14.855913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.855837    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8" pod "57fc6fe4-3676-42d5-834f-688b6cf2365c" (UID: "57fc6fe4-3676-42d5-834f-688b6cf2365c"). InnerVolumeSpecName "kube-api-access-v4pg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:14.955000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.954956    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:14.955000 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.954996    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fc6fe4-3676-42d5-834f-688b6cf2365c-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:14.955201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:14.955010    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pg8\" (UniqueName: \"kubernetes.io/projected/57fc6fe4-3676-42d5-834f-688b6cf2365c-kube-api-access-v4pg8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:15.581872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:15.581842    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"
Oct 30 03:34:15.582063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:15.581845    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z" event={"ID":"57fc6fe4-3676-42d5-834f-688b6cf2365c","Type":"ContainerDied","Data":"74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d"}
Oct 30 03:34:15.582063 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:15.581963    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e828dc49fb353f8707ab5e9edff4cef5a5b1439eaeb32bb836062192cdab8d"
Oct 30 03:34:31.339523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.339475    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"]
Oct 30 03:34:31.340008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.339894    4214 cpu_manager.go:414] "containerMap: removing container" podUID="57fc6fe4-3676-42d5-834f-688b6cf2365c" containerName="revoke-keys"
Oct 30 03:34:31.340008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.339905    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fc6fe4-3676-42d5-834f-688b6cf2365c" containerName="revoke-keys"
Oct 30 03:34:31.340008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.339995    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fc6fe4-3676-42d5-834f-688b6cf2365c" containerName="revoke-keys"
Oct 30 03:34:31.343121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.343104    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.361309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.361282    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"]
Oct 30 03:34:31.371313 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.371289    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:34:31.371450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.371382    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:34:31.425536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.425498    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.425536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.425541    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxp8\" (UniqueName: \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.425767 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.425636    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.526525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.526493    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.526726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.526536    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxp8\" (UniqueName: \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.526726 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.526590    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.527339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.527313    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.528920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.528894    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.546178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.546151    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxp8\" (UniqueName: \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\") pod \"maas-api-revoke-sub-2e1f5b5f-8vjx7\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.655476 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.655378    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:31.956014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:31.955990    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"]
Oct 30 03:34:31.958465 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:34:31.958430    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa57a4cd_6c66_4933_a210_f7e840452ab4.slice/crio-2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71 WatchSource:0}: Error finding container 2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71: Status 404 returned error can't find the container with id 2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71
Oct 30 03:34:32.694205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:32.694160    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7" event={"ID":"aa57a4cd-6c66-4933-a210-f7e840452ab4","Type":"ContainerStarted","Data":"7f5ddc74de174e726ad3614cf8594083cbb9a7b9aa7a237a3b6c887f02f60be4"}
Oct 30 03:34:32.694600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:32.694216    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7" event={"ID":"aa57a4cd-6c66-4933-a210-f7e840452ab4","Type":"ContainerStarted","Data":"2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71"}
Oct 30 03:34:32.735711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:32.735659    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7" podStartSLOduration=1.735644757 podStartE2EDuration="1.735644757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:32.733459103 +0000 UTC m=+3211.807629895" watchObservedRunningTime="2026-10-30 03:34:32.735644757 +0000 UTC m=+3211.809815549"
Oct 30 03:34:33.700957 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:33.700917    4214 generic.go:356] "Generic (PLEG): container finished" podID="aa57a4cd-6c66-4933-a210-f7e840452ab4" containerID="7f5ddc74de174e726ad3614cf8594083cbb9a7b9aa7a237a3b6c887f02f60be4" exitCode=0
Oct 30 03:34:33.701375 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:33.701009    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7" event={"ID":"aa57a4cd-6c66-4933-a210-f7e840452ab4","Type":"ContainerDied","Data":"7f5ddc74de174e726ad3614cf8594083cbb9a7b9aa7a237a3b6c887f02f60be4"}
Oct 30 03:34:34.831830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.831804    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:34.964386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.964303    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\") pod \"aa57a4cd-6c66-4933-a210-f7e840452ab4\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") "
Oct 30 03:34:34.964571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.964401    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\") pod \"aa57a4cd-6c66-4933-a210-f7e840452ab4\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") "
Oct 30 03:34:34.964571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.964448    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\" (UniqueName: \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\") pod \"aa57a4cd-6c66-4933-a210-f7e840452ab4\" (UID: \"aa57a4cd-6c66-4933-a210-f7e840452ab4\") "
Oct 30 03:34:34.964716 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.964685    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca" pod "aa57a4cd-6c66-4933-a210-f7e840452ab4" (UID: "aa57a4cd-6c66-4933-a210-f7e840452ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:34.966549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.966520    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8" pod "aa57a4cd-6c66-4933-a210-f7e840452ab4" (UID: "aa57a4cd-6c66-4933-a210-f7e840452ab4"). InnerVolumeSpecName "kube-api-access-8fxp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.966549 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:34.966535    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca" pod "aa57a4cd-6c66-4933-a210-f7e840452ab4" (UID: "aa57a4cd-6c66-4933-a210-f7e840452ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:35.065177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.065146    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:35.065177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.065173    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa57a4cd-6c66-4933-a210-f7e840452ab4-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:35.065177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.065181    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxp8\" (UniqueName: \"kubernetes.io/projected/aa57a4cd-6c66-4933-a210-f7e840452ab4-kube-api-access-8fxp8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:35.441896 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.441748    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"]
Oct 30 03:34:35.442551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.442523    4214 cpu_manager.go:414] "containerMap: removing container" podUID="aa57a4cd-6c66-4933-a210-f7e840452ab4" containerName="revoke-keys"
Oct 30 03:34:35.442551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.442546    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa57a4cd-6c66-4933-a210-f7e840452ab4" containerName="revoke-keys"
Oct 30 03:34:35.442701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.442631    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa57a4cd-6c66-4933-a210-f7e840452ab4" containerName="revoke-keys"
Oct 30 03:34:35.445720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.445703    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.478599 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.478564    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"]
Oct 30 03:34:35.568963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.568924    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.568963 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.568967    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569056    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569097    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmpm\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-kube-api-access-jqmpm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569173    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569217    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569297    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569336    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.569449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.569394    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.669927 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.669890    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.669946    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670076    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670140    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670174    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmpm\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-kube-api-access-jqmpm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670220    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670308 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670289    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670335    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670399    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670418    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670443    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670724    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.670789 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.670750    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.671074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.671051    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.672597 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.672576    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.672744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.672724    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.690627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.690591    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.690752 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.690684    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmpm\" (UniqueName: \"kubernetes.io/projected/0a13416b-810d-4f7c-9aa8-e3809e65ccfa-kube-api-access-jqmpm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl5mn\" (UID: \"0a13416b-810d-4f7c-9aa8-e3809e65ccfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:35.712376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.712306    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"
Oct 30 03:34:35.712376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.712325    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7" event={"ID":"aa57a4cd-6c66-4933-a210-f7e840452ab4","Type":"ContainerDied","Data":"2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71"}
Oct 30 03:34:35.712376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.712371    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d566fc40bf856a4a8b7c0563851e2c37b12ec79c4ceba6609fd979b9bd35f71"
Oct 30 03:34:35.759665 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:35.759638    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:36.091316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.091290    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"]
Oct 30 03:34:36.091929 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:34:36.091902    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a13416b_810d_4f7c_9aa8_e3809e65ccfa.slice/crio-dbe90f39a5529db08f476ddac839d544f340b5a6eeddd64eaabc274931387c42 WatchSource:0}: Error finding container dbe90f39a5529db08f476ddac839d544f340b5a6eeddd64eaabc274931387c42: Status 404 returned error can't find the container with id dbe90f39a5529db08f476ddac839d544f340b5a6eeddd64eaabc274931387c42
Oct 30 03:34:36.094163 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.094130    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.094282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.094190    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.094282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.094221    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:34:36.719924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.719888    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" event={"ID":"0a13416b-810d-4f7c-9aa8-e3809e65ccfa","Type":"ContainerStarted","Data":"081129c3af0a08fccb3937ead33cdf203c9af997dc2f394b2a3aa69dc3549cbf"}
Oct 30 03:34:36.720122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.719941    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" event={"ID":"0a13416b-810d-4f7c-9aa8-e3809e65ccfa","Type":"ContainerStarted","Data":"dbe90f39a5529db08f476ddac839d544f340b5a6eeddd64eaabc274931387c42"}
Oct 30 03:34:36.759979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.759939    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:36.802318 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:36.802259    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podStartSLOduration=1.8022454140000002 podStartE2EDuration="1.802245414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:36.787914843 +0000 UTC m=+3215.862085640" watchObservedRunningTime="2026-10-30 03:34:36.802245414 +0000 UTC m=+3215.876416246"
Oct 30 03:34:37.762190 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:37.762156    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:37.762624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:37.762213    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:38.761015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:38.760974    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:38.761264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:38.761033    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:39.184688 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:34:39.184652    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:34:39.760706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:39.760668    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:39.760904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:39.760722    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:40.761256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:40.760863    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:40.761256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:40.760926    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:41.760720 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:41.760677    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:41.760903 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:41.760764    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:42.617275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.617218    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"]
Oct 30 03:34:42.621475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.621449    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.646585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.646550    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:34:42.646758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.646658    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:34:42.664460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.664426    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"]
Oct 30 03:34:42.739883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.739848    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.740082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.739901    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765bd\" (UniqueName: \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.740082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.740049    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.761007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.760939    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 03:34:42.761205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.761038    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 03:34:42.840967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.840929    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.841197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.840998    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.841197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.841063    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765bd\" (UniqueName: \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.841966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.841932    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.843826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.843806    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.865594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.865570    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765bd\" (UniqueName: \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\") pod \"maas-api-revoke-sub-ec843018-p97ql\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:42.939491 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:42.939427    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:43.257573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:43.257547    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"]
Oct 30 03:34:43.261069 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:34:43.261039    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fbaa96_ff8c_42f6_9014_4782a8ca04ed.slice/crio-004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe WatchSource:0}: Error finding container 004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe: Status 404 returned error can't find the container with id 004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe
Oct 30 03:34:43.783029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:43.782986    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql" event={"ID":"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed","Type":"ContainerStarted","Data":"c06f2c4b2db5059c5a209d0ae314309e4d85b2bdc384c159e9a47ae0a8445f35"}
Oct 30 03:34:43.783557 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:43.783057    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql" event={"ID":"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed","Type":"ContainerStarted","Data":"004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe"}
Oct 30 03:34:43.825659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:43.825592    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql" podStartSLOduration=1.825573804 podStartE2EDuration="1.825573804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:43.823549124 +0000 UTC m=+3222.897719918" watchObservedRunningTime="2026-10-30 03:34:43.825573804 +0000 UTC m=+3222.899744598"
Oct 30 03:34:44.761834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:44.761786    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:44.762044 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:44.761867    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:44.791856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:44.791821    4214 generic.go:356] "Generic (PLEG): container finished" podID="b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" containerID="c06f2c4b2db5059c5a209d0ae314309e4d85b2bdc384c159e9a47ae0a8445f35" exitCode=0
Oct 30 03:34:44.792277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:44.791918    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql" event={"ID":"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed","Type":"ContainerDied","Data":"c06f2c4b2db5059c5a209d0ae314309e4d85b2bdc384c159e9a47ae0a8445f35"}
Oct 30 03:34:45.804429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:45.804391    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:45.804769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:45.804475    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn" podUID="0a13416b-810d-4f7c-9aa8-e3809e65ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:45.960945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:45.960918    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:46.085451 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.085359    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\" (UniqueName: \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\") pod \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") "
Oct 30 03:34:46.085646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.085467    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\") pod \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") "
Oct 30 03:34:46.085646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.085507    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\") pod \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\" (UID: \"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed\") "
Oct 30 03:34:46.086010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.085981    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca" pod "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" (UID: "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:46.087784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.087755    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca" pod "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" (UID: "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:46.089054 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.089025    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd" pod "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" (UID: "b0fbaa96-ff8c-42f6-9014-4782a8ca04ed"). InnerVolumeSpecName "kube-api-access-765bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:46.186801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.186765    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765bd\" (UniqueName: \"kubernetes.io/projected/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-kube-api-access-765bd\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:46.186801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.186797    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:46.186801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.186807    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:34:46.708439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.708409    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:46.708696 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.708673    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:46.709300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.709282    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl5mn"
Oct 30 03:34:46.804555 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.804528    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"
Oct 30 03:34:46.805002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.804573    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql" event={"ID":"b0fbaa96-ff8c-42f6-9014-4782a8ca04ed","Type":"ContainerDied","Data":"004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe"}
Oct 30 03:34:46.805002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:34:46.804629    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004c5b298816493f676874363a2a981ac9c87033d1e9deb9931842ff9499a5fe"
Oct 30 03:35:09.163922 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:35:09.163884    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:35:10.671065 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.671032    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"]
Oct 30 03:35:10.671518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.671503    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" containerName="revoke-keys"
Oct 30 03:35:10.671569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.671519    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" containerName="revoke-keys"
Oct 30 03:35:10.671606 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.671588    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" containerName="revoke-keys"
Oct 30 03:35:10.674865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.674842    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.692743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.692719    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:35:10.705329 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.705293    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:35:10.708747 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.708721    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"]
Oct 30 03:35:10.791160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.791127    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgxb\" (UniqueName: \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.791339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.791173    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.791339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.791323    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.892047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.892013    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgxb\" (UniqueName: \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.892278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.892056    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.892278 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.892116    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.892750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.892727    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.894388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.894363    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.912546 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.912517    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgxb\" (UniqueName: \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\") pod \"maas-api-revoke-sub-046e1343-722c4\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:10.988004 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:10.987935    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:11.288253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:11.288209    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"]
Oct 30 03:35:11.290318 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:35:11.290285    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d42ddd_3dd0_4c9a_93fd_8c8644afcbed.slice/crio-d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6 WatchSource:0}: Error finding container d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6: Status 404 returned error can't find the container with id d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6
Oct 30 03:35:11.973664 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:11.973624    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4" event={"ID":"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed","Type":"ContainerStarted","Data":"5b30f30f324ecf303f26a2fbc6b6434df1684c972e6233902b80cdc404cf9502"}
Oct 30 03:35:11.974069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:11.973691    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4" event={"ID":"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed","Type":"ContainerStarted","Data":"d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6"}
Oct 30 03:35:12.017807 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:12.017759    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4" podStartSLOduration=2.017745173 podStartE2EDuration="2.017745173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:12.015748648 +0000 UTC m=+3251.089919441" watchObservedRunningTime="2026-10-30 03:35:12.017745173 +0000 UTC m=+3251.091915966"
Oct 30 03:35:12.980840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:12.980803    4214 generic.go:356] "Generic (PLEG): container finished" podID="14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" containerID="5b30f30f324ecf303f26a2fbc6b6434df1684c972e6233902b80cdc404cf9502" exitCode=0
Oct 30 03:35:12.981223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:12.980857    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4" event={"ID":"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed","Type":"ContainerDied","Data":"5b30f30f324ecf303f26a2fbc6b6434df1684c972e6233902b80cdc404cf9502"}
Oct 30 03:35:14.124499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.124472    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:14.227526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.227492    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\" (UniqueName: \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\") pod \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") "
Oct 30 03:35:14.227709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.227545    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\") pod \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") "
Oct 30 03:35:14.227709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.227577    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\") pod \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\" (UID: \"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed\") "
Oct 30 03:35:14.227921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.227898    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca" pod "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" (UID: "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:14.229533 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.229508    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca" pod "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" (UID: "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:14.229648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.229607    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb" pod "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" (UID: "14d42ddd-3dd0-4c9a-93fd-8c8644afcbed"). InnerVolumeSpecName "kube-api-access-wbgxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:14.328517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.328489    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:14.328517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.328513    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgxb\" (UniqueName: \"kubernetes.io/projected/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-kube-api-access-wbgxb\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:14.328687 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.328523    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:14.992843 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.992812    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"
Oct 30 03:35:14.993020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.992844    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4" event={"ID":"14d42ddd-3dd0-4c9a-93fd-8c8644afcbed","Type":"ContainerDied","Data":"d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6"}
Oct 30 03:35:14.993020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:14.992896    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4cd823e0c05a4094ec94853ba8ac01acee018757912212d348c9f778f017ff6"
Oct 30 03:35:21.025613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.025577    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"]
Oct 30 03:35:21.026188 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.026172    4214 cpu_manager.go:414] "containerMap: removing container" podUID="14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" containerName="revoke-keys"
Oct 30 03:35:21.026251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.026190    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" containerName="revoke-keys"
Oct 30 03:35:21.026324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.026313    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" containerName="revoke-keys"
Oct 30 03:35:21.029895 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.029868    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.046865 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.046834    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"]
Oct 30 03:35:21.056199 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.056168    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:35:21.056371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.056282    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:35:21.092494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.092448    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.092676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.092584    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.092676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.092671    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6tl\" (UniqueName: \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.194309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.194262    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6tl\" (UniqueName: \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.194513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.194373    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.194513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.194432    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.195162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.195136    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.196947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.196928    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.214592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.214566    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6tl\" (UniqueName: \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\") pod \"maas-api-revoke-sub-d3fba056-lkltj\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.344576 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.344535    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:21.663947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:21.663916    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"]
Oct 30 03:35:21.667461 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:35:21.667434    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa452646_8139_47a4_89ae_153a7b7eee1b.slice/crio-8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9 WatchSource:0}: Error finding container 8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9: Status 404 returned error can't find the container with id 8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9
Oct 30 03:35:22.035694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:22.035657    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj" event={"ID":"fa452646-8139-47a4-89ae-153a7b7eee1b","Type":"ContainerStarted","Data":"033f84b0393eaa5f2449c1d43561bba5a39cf1fc8c7f4540c689add6d5dd0c92"}
Oct 30 03:35:22.036122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:22.035749    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj" event={"ID":"fa452646-8139-47a4-89ae-153a7b7eee1b","Type":"ContainerStarted","Data":"8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9"}
Oct 30 03:35:22.080410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:22.080361    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj" podStartSLOduration=1.080344695 podStartE2EDuration="1.080344695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:22.077436871 +0000 UTC m=+3261.151607675" watchObservedRunningTime="2026-10-30 03:35:22.080344695 +0000 UTC m=+3261.154515489"
Oct 30 03:35:23.044693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:23.044661    4214 generic.go:356] "Generic (PLEG): container finished" podID="fa452646-8139-47a4-89ae-153a7b7eee1b" containerID="033f84b0393eaa5f2449c1d43561bba5a39cf1fc8c7f4540c689add6d5dd0c92" exitCode=0
Oct 30 03:35:23.045112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:23.044754    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj" event={"ID":"fa452646-8139-47a4-89ae-153a7b7eee1b","Type":"ContainerDied","Data":"033f84b0393eaa5f2449c1d43561bba5a39cf1fc8c7f4540c689add6d5dd0c92"}
Oct 30 03:35:24.586540 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.586508    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:24.658251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.658202    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\" (UniqueName: \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\") pod \"fa452646-8139-47a4-89ae-153a7b7eee1b\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") "
Oct 30 03:35:24.658465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.658287    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\") pod \"fa452646-8139-47a4-89ae-153a7b7eee1b\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") "
Oct 30 03:35:24.658465 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.658426    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\") pod \"fa452646-8139-47a4-89ae-153a7b7eee1b\" (UID: \"fa452646-8139-47a4-89ae-153a7b7eee1b\") "
Oct 30 03:35:24.665251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.664788    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca" pod "fa452646-8139-47a4-89ae-153a7b7eee1b" (UID: "fa452646-8139-47a4-89ae-153a7b7eee1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:24.675528 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.675475    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca" pod "fa452646-8139-47a4-89ae-153a7b7eee1b" (UID: "fa452646-8139-47a4-89ae-153a7b7eee1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:24.695906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.695602    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl" pod "fa452646-8139-47a4-89ae-153a7b7eee1b" (UID: "fa452646-8139-47a4-89ae-153a7b7eee1b"). InnerVolumeSpecName "kube-api-access-fq6tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:24.760171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.759993    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:24.760171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.760028    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6tl\" (UniqueName: \"kubernetes.io/projected/fa452646-8139-47a4-89ae-153a7b7eee1b-kube-api-access-fq6tl\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:24.760171 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:24.760041    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa452646-8139-47a4-89ae-153a7b7eee1b-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:25.081693 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:25.081667    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"
Oct 30 03:35:25.081849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:25.081670    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj" event={"ID":"fa452646-8139-47a4-89ae-153a7b7eee1b","Type":"ContainerDied","Data":"8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9"}
Oct 30 03:35:25.081849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:25.081786    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0feb3cf4d7787cd66b4208f60e640761614adef0eb29a6d35cc22572743ca9"
Oct 30 03:35:39.223427 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:35:39.223329    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:35:42.559763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.559720    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"]
Oct 30 03:35:42.560205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.560116    4214 cpu_manager.go:414] "containerMap: removing container" podUID="fa452646-8139-47a4-89ae-153a7b7eee1b" containerName="revoke-keys"
Oct 30 03:35:42.560205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.560127    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa452646-8139-47a4-89ae-153a7b7eee1b" containerName="revoke-keys"
Oct 30 03:35:42.560324 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.560205    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa452646-8139-47a4-89ae-153a7b7eee1b" containerName="revoke-keys"
Oct 30 03:35:42.563532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.563516    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.589648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.589617    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:35:42.589810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.589678    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:35:42.594185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.594159    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"]
Oct 30 03:35:42.659296 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.659253    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnwc\" (UniqueName: \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.659488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.659308    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.659488 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.659458    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.760196 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.760163    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.760422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.760269    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.760422 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.760312    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnwc\" (UniqueName: \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.760981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.760952    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.762511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.762481    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.780428 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.780402    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnwc\" (UniqueName: \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\") pod \"maas-api-revoke-sub-ae594979-kx9tq\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:42.876852 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:42.876762    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:43.177874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:43.177848    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"]
Oct 30 03:35:43.178872 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:35:43.178842    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1c3afc_ce80_4e32_a686_ce6226f4d71a.slice/crio-4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad WatchSource:0}: Error finding container 4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad: Status 404 returned error can't find the container with id 4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad
Oct 30 03:35:43.217107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:43.217073    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq" event={"ID":"cc1c3afc-ce80-4e32-a686-ce6226f4d71a","Type":"ContainerStarted","Data":"4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad"}
Oct 30 03:35:44.224025 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:44.223941    4214 generic.go:356] "Generic (PLEG): container finished" podID="cc1c3afc-ce80-4e32-a686-ce6226f4d71a" containerID="95311ef351d8bbe1df49c30d01a19a7193cd922d0cc37a77ec904f586726debe" exitCode=0
Oct 30 03:35:44.224410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:44.224025    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq" event={"ID":"cc1c3afc-ce80-4e32-a686-ce6226f4d71a","Type":"ContainerDied","Data":"95311ef351d8bbe1df49c30d01a19a7193cd922d0cc37a77ec904f586726debe"}
Oct 30 03:35:45.360830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.360806    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:45.382699 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.382676    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\") pod \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") "
Oct 30 03:35:45.382820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.382714    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\") pod \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") "
Oct 30 03:35:45.382820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.382784    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\" (UniqueName: \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\") pod \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\" (UID: \"cc1c3afc-ce80-4e32-a686-ce6226f4d71a\") "
Oct 30 03:35:45.383144 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.383123    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca" pod "cc1c3afc-ce80-4e32-a686-ce6226f4d71a" (UID: "cc1c3afc-ce80-4e32-a686-ce6226f4d71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:45.384744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.384719    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca" pod "cc1c3afc-ce80-4e32-a686-ce6226f4d71a" (UID: "cc1c3afc-ce80-4e32-a686-ce6226f4d71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:45.385161 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.385131    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc" pod "cc1c3afc-ce80-4e32-a686-ce6226f4d71a" (UID: "cc1c3afc-ce80-4e32-a686-ce6226f4d71a"). InnerVolumeSpecName "kube-api-access-qgnwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:45.484569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.484491    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:45.484569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.484518    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnwc\" (UniqueName: \"kubernetes.io/projected/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-kube-api-access-qgnwc\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:45.484569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:45.484529    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c3afc-ce80-4e32-a686-ce6226f4d71a-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:35:46.236116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:46.236083    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq" event={"ID":"cc1c3afc-ce80-4e32-a686-ce6226f4d71a","Type":"ContainerDied","Data":"4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad"}
Oct 30 03:35:46.236116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:46.236118    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"
Oct 30 03:35:46.236354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:35:46.236131    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4426d072d37bbd7deb6c3aa3900bc726d47d5878be5f42882940e15a5d944aad"
Oct 30 03:36:04.191373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.191340    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"]
Oct 30 03:36:04.191812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.191766    4214 cpu_manager.go:414] "containerMap: removing container" podUID="cc1c3afc-ce80-4e32-a686-ce6226f4d71a" containerName="revoke-keys"
Oct 30 03:36:04.191812 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.191778    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1c3afc-ce80-4e32-a686-ce6226f4d71a" containerName="revoke-keys"
Oct 30 03:36:04.191894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.191873    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1c3afc-ce80-4e32-a686-ce6226f4d71a" containerName="revoke-keys"
Oct 30 03:36:04.195253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.195222    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.214632 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.214605    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"]
Oct 30 03:36:04.223349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.223321    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:36:04.223521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.223403    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:36:04.275405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.275375    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.275568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.275502    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.275613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.275575    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfb7\" (UniqueName: \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.376305 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.376271    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.376479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.376354    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfb7\" (UniqueName: \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.376479 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.376391    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.377067 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.377042    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.378658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.378632    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.396978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.396954    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfb7\" (UniqueName: \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\") pod \"maas-api-revoke-sub-a4aef7d7-gh6ks\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.508215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.508145    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:04.807535 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:04.807502    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"]
Oct 30 03:36:04.809061 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:36:04.809035    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b3daa7_02d3_4652_8d2f_881d3b42f121.slice/crio-033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942 WatchSource:0}: Error finding container 033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942: Status 404 returned error can't find the container with id 033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942
Oct 30 03:36:05.385511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:05.385476    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks" event={"ID":"20b3daa7-02d3-4652-8d2f-881d3b42f121","Type":"ContainerStarted","Data":"cb35e727b9c93ffd8ff2ef0fd076952d5e4c74d7876f220482c8332acdd42802"}
Oct 30 03:36:05.385915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:05.385542    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks" event={"ID":"20b3daa7-02d3-4652-8d2f-881d3b42f121","Type":"ContainerStarted","Data":"033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942"}
Oct 30 03:36:05.426282 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:05.426178    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks" podStartSLOduration=1.426163569 podStartE2EDuration="1.426163569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:05.424833469 +0000 UTC m=+3304.499004262" watchObservedRunningTime="2026-10-30 03:36:05.426163569 +0000 UTC m=+3304.500334391"
Oct 30 03:36:06.391968 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:06.391932    4214 generic.go:356] "Generic (PLEG): container finished" podID="20b3daa7-02d3-4652-8d2f-881d3b42f121" containerID="cb35e727b9c93ffd8ff2ef0fd076952d5e4c74d7876f220482c8332acdd42802" exitCode=0
Oct 30 03:36:06.392405 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:06.392019    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks" event={"ID":"20b3daa7-02d3-4652-8d2f-881d3b42f121","Type":"ContainerDied","Data":"cb35e727b9c93ffd8ff2ef0fd076952d5e4c74d7876f220482c8332acdd42802"}
Oct 30 03:36:07.535586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.535561    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:07.606170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.606143    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\") pod \"20b3daa7-02d3-4652-8d2f-881d3b42f121\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") "
Oct 30 03:36:07.606342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.606256    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\") pod \"20b3daa7-02d3-4652-8d2f-881d3b42f121\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") "
Oct 30 03:36:07.606392 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.606347    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\" (UniqueName: \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\") pod \"20b3daa7-02d3-4652-8d2f-881d3b42f121\" (UID: \"20b3daa7-02d3-4652-8d2f-881d3b42f121\") "
Oct 30 03:36:07.606692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.606660    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca" pod "20b3daa7-02d3-4652-8d2f-881d3b42f121" (UID: "20b3daa7-02d3-4652-8d2f-881d3b42f121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:07.608406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.608383    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7" pod "20b3daa7-02d3-4652-8d2f-881d3b42f121" (UID: "20b3daa7-02d3-4652-8d2f-881d3b42f121"). InnerVolumeSpecName "kube-api-access-7mfb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:07.608494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.608425    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca" pod "20b3daa7-02d3-4652-8d2f-881d3b42f121" (UID: "20b3daa7-02d3-4652-8d2f-881d3b42f121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:07.707333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.707269    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:07.707333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.707294    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b3daa7-02d3-4652-8d2f-881d3b42f121-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:07.707333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:07.707306    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfb7\" (UniqueName: \"kubernetes.io/projected/20b3daa7-02d3-4652-8d2f-881d3b42f121-kube-api-access-7mfb7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:08.405213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:08.405185    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"
Oct 30 03:36:08.405406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:08.405222    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks" event={"ID":"20b3daa7-02d3-4652-8d2f-881d3b42f121","Type":"ContainerDied","Data":"033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942"}
Oct 30 03:36:08.405406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:08.405291    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033149a9ea4feb87987f936b3deb1b2e9e301fbe20b636d3e5302fb9673fd942"
Oct 30 03:36:09.164910 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:36:09.164873    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:36:25.371322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.371282    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"]
Oct 30 03:36:25.371967 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.371943    4214 cpu_manager.go:414] "containerMap: removing container" podUID="20b3daa7-02d3-4652-8d2f-881d3b42f121" containerName="revoke-keys"
Oct 30 03:36:25.372018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.371972    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b3daa7-02d3-4652-8d2f-881d3b42f121" containerName="revoke-keys"
Oct 30 03:36:25.372134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.372120    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b3daa7-02d3-4652-8d2f-881d3b42f121" containerName="revoke-keys"
Oct 30 03:36:25.375819 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.375800    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.392525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.392493    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:36:25.392810 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.392509    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:36:25.408888 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.408857    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"]
Oct 30 03:36:25.540399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.540368    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dswz\" (UniqueName: \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.540613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.540426    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.540613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.540560    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.641212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.641114    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dswz\" (UniqueName: \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.641212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.641179    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.641519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.641253    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.641853 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.641827    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.643564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.643546    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.663695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.663668    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dswz\" (UniqueName: \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\") pod \"maas-api-revoke-sub-79805ca9-pzmh8\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.690517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.690489    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:25.994265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:25.994213    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"]
Oct 30 03:36:25.995780 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:36:25.995754    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8d0d45_ae73_410a_8a6e_f7db870ffa0c.slice/crio-481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4 WatchSource:0}: Error finding container 481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4: Status 404 returned error can't find the container with id 481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4
Oct 30 03:36:26.536757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:26.536670    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8" event={"ID":"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c","Type":"ContainerStarted","Data":"6cca668307d7ba32507c0d2064e8aa36c17f9f4a30e3874e4494284b3d4e3907"}
Oct 30 03:36:26.536757 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:26.536726    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8" event={"ID":"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c","Type":"ContainerStarted","Data":"481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4"}
Oct 30 03:36:26.581637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:26.581586    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8" podStartSLOduration=1.581572443 podStartE2EDuration="1.581572443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:26.580955612 +0000 UTC m=+3325.655126416" watchObservedRunningTime="2026-10-30 03:36:26.581572443 +0000 UTC m=+3325.655743257"
Oct 30 03:36:27.543587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:27.543552    4214 generic.go:356] "Generic (PLEG): container finished" podID="2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" containerID="6cca668307d7ba32507c0d2064e8aa36c17f9f4a30e3874e4494284b3d4e3907" exitCode=0
Oct 30 03:36:27.543996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:27.543603    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8" event={"ID":"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c","Type":"ContainerDied","Data":"6cca668307d7ba32507c0d2064e8aa36c17f9f4a30e3874e4494284b3d4e3907"}
Oct 30 03:36:28.674307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.674279    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:28.771568 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.771530    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\") pod \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") "
Oct 30 03:36:28.771780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.771632    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\" (UniqueName: \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\") pod \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") "
Oct 30 03:36:28.771780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.771762    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\") pod \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\" (UID: \"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c\") "
Oct 30 03:36:28.771937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.771900    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca" pod "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" (UID: "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:28.772126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.772109    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:28.773730 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.773706    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz" pod "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" (UID: "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c"). InnerVolumeSpecName "kube-api-access-4dswz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:28.773828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.773726    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca" pod "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" (UID: "2c8d0d45-ae73-410a-8a6e-f7db870ffa0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:28.873123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.873038    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dswz\" (UniqueName: \"kubernetes.io/projected/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-kube-api-access-4dswz\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:28.873123 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:28.873070    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.555953 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:29.555922    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"
Oct 30 03:36:29.556138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:29.555958    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8" event={"ID":"2c8d0d45-ae73-410a-8a6e-f7db870ffa0c","Type":"ContainerDied","Data":"481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4"}
Oct 30 03:36:29.556138 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:29.556011    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="481e9aae5fa6ae5fab29b858ed9eef91832fa03353e7ce7c18346e10d48aeac4"
Oct 30 03:36:39.168669 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:36:39.168612    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:36:46.557541 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.557503    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"]
Oct 30 03:36:46.557974 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.557959    4214 cpu_manager.go:414] "containerMap: removing container" podUID="2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" containerName="revoke-keys"
Oct 30 03:36:46.558020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.557975    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" containerName="revoke-keys"
Oct 30 03:36:46.558056 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.558036    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" containerName="revoke-keys"
Oct 30 03:36:46.561215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.561196    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.580464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.580431    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"]
Oct 30 03:36:46.589087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.589054    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:36:46.589087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.589083    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:36:46.683815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.683768    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.683815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.683812    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.684078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.683889    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5gk\" (UniqueName: \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.785068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.785027    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5gk\" (UniqueName: \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.785287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.785135    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.785287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.785156    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.785837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.785817    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.787594 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.787568    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.805471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.805444    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5gk\" (UniqueName: \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\") pod \"maas-api-revoke-sub-0b7e8edb-s6z4x\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:46.874455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:46.874370    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:47.175837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:47.175808    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"]
Oct 30 03:36:47.177596 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:36:47.177570    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb11e4e2_9d17_475c_bd94_2e044c6539e5.slice/crio-65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c WatchSource:0}: Error finding container 65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c: Status 404 returned error can't find the container with id 65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c
Oct 30 03:36:47.664610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:47.664571    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x" event={"ID":"db11e4e2-9d17-475c-bd94-2e044c6539e5","Type":"ContainerStarted","Data":"f26a39a67563d953fcff96747288643508da0cd4d7ebf520a99896aceb864370"}
Oct 30 03:36:47.664993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:47.664633    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x" event={"ID":"db11e4e2-9d17-475c-bd94-2e044c6539e5","Type":"ContainerStarted","Data":"65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c"}
Oct 30 03:36:47.706702 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:47.706601    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x" podStartSLOduration=1.706587239 podStartE2EDuration="1.706587239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:47.704052318 +0000 UTC m=+3346.778223124" watchObservedRunningTime="2026-10-30 03:36:47.706587239 +0000 UTC m=+3346.780758031"
Oct 30 03:36:48.671744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:48.671704    4214 generic.go:356] "Generic (PLEG): container finished" podID="db11e4e2-9d17-475c-bd94-2e044c6539e5" containerID="f26a39a67563d953fcff96747288643508da0cd4d7ebf520a99896aceb864370" exitCode=0
Oct 30 03:36:48.672150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:48.671793    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x" event={"ID":"db11e4e2-9d17-475c-bd94-2e044c6539e5","Type":"ContainerDied","Data":"f26a39a67563d953fcff96747288643508da0cd4d7ebf520a99896aceb864370"}
Oct 30 03:36:49.815851 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.815818    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:49.916827 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.916796    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\") pod \"db11e4e2-9d17-475c-bd94-2e044c6539e5\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") "
Oct 30 03:36:49.917014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.916843    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\" (UniqueName: \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\") pod \"db11e4e2-9d17-475c-bd94-2e044c6539e5\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") "
Oct 30 03:36:49.917014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.916956    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\") pod \"db11e4e2-9d17-475c-bd94-2e044c6539e5\" (UID: \"db11e4e2-9d17-475c-bd94-2e044c6539e5\") "
Oct 30 03:36:49.917357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.917324    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca" pod "db11e4e2-9d17-475c-bd94-2e044c6539e5" (UID: "db11e4e2-9d17-475c-bd94-2e044c6539e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:49.918934 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.918913    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca" pod "db11e4e2-9d17-475c-bd94-2e044c6539e5" (UID: "db11e4e2-9d17-475c-bd94-2e044c6539e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:49.919029 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:49.919007    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk" pod "db11e4e2-9d17-475c-bd94-2e044c6539e5" (UID: "db11e4e2-9d17-475c-bd94-2e044c6539e5"). InnerVolumeSpecName "kube-api-access-sq5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:50.017959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.017883    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:50.017959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.017914    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5gk\" (UniqueName: \"kubernetes.io/projected/db11e4e2-9d17-475c-bd94-2e044c6539e5-kube-api-access-sq5gk\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:50.017959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.017927    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db11e4e2-9d17-475c-bd94-2e044c6539e5-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:36:50.684731 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.684702    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"
Oct 30 03:36:50.684892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.684731    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x" event={"ID":"db11e4e2-9d17-475c-bd94-2e044c6539e5","Type":"ContainerDied","Data":"65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c"}
Oct 30 03:36:50.684892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:50.684772    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c51cff06d5907f0b1d7643b1c5782140d9d579a78f697095ce22d8b25e2e8c"
Oct 30 03:36:57.460790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.460746    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"]
Oct 30 03:36:57.461490 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.461464    4214 cpu_manager.go:414] "containerMap: removing container" podUID="db11e4e2-9d17-475c-bd94-2e044c6539e5" containerName="revoke-keys"
Oct 30 03:36:57.461539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.461493    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db11e4e2-9d17-475c-bd94-2e044c6539e5" containerName="revoke-keys"
Oct 30 03:36:57.461620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.461606    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="db11e4e2-9d17-475c-bd94-2e044c6539e5" containerName="revoke-keys"
Oct 30 03:36:57.465122 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.465097    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.483685 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.483657    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"]
Oct 30 03:36:57.490922 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.490895    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:36:57.491078 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.491058    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:36:57.589948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.589908    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9m74\" (UniqueName: \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.589948 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.589946    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.590212 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.589971    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.690725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.690686    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.690926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.690733    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.690926 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.690891    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9m74\" (UniqueName: \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.691515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.691489    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.693155 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.693129    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.725460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.725401    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9m74\" (UniqueName: \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\") pod \"maas-api-revoke-sub-3d4156f1-m6cpr\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:57.779061 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:57.779035    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:36:58.092900 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:58.092871    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"]
Oct 30 03:36:58.094085 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:36:58.094061    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07d84bb_19ce_4d24_bee7_264d4b85ae22.slice/crio-04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f WatchSource:0}: Error finding container 04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f: Status 404 returned error can't find the container with id 04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f
Oct 30 03:36:58.730164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:58.730071    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr" event={"ID":"b07d84bb-19ce-4d24-bee7-264d4b85ae22","Type":"ContainerStarted","Data":"56fdf20c3e26feb17cc4b2a41b6da011be1e16cfca51b5c2a45cbaec944e6d9a"}
Oct 30 03:36:58.730164 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:58.730125    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr" event={"ID":"b07d84bb-19ce-4d24-bee7-264d4b85ae22","Type":"ContainerStarted","Data":"04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f"}
Oct 30 03:36:58.776356 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:58.776298    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr" podStartSLOduration=1.776284692 podStartE2EDuration="1.776284692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:58.775099966 +0000 UTC m=+3357.849270760" watchObservedRunningTime="2026-10-30 03:36:58.776284692 +0000 UTC m=+3357.850455485"
Oct 30 03:36:59.737738 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:59.737701    4214 generic.go:356] "Generic (PLEG): container finished" podID="b07d84bb-19ce-4d24-bee7-264d4b85ae22" containerID="56fdf20c3e26feb17cc4b2a41b6da011be1e16cfca51b5c2a45cbaec944e6d9a" exitCode=0
Oct 30 03:36:59.738150 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:36:59.737759    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr" event={"ID":"b07d84bb-19ce-4d24-bee7-264d4b85ae22","Type":"ContainerDied","Data":"56fdf20c3e26feb17cc4b2a41b6da011be1e16cfca51b5c2a45cbaec944e6d9a"}
Oct 30 03:37:00.893217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:00.893186    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:37:01.022658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.022574    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\") pod \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") "
Oct 30 03:37:01.022815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.022671    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\" (UniqueName: \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\") pod \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") "
Oct 30 03:37:01.022815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.022776    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\") pod \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\" (UID: \"b07d84bb-19ce-4d24-bee7-264d4b85ae22\") "
Oct 30 03:37:01.023082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.023057    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca" pod "b07d84bb-19ce-4d24-bee7-264d4b85ae22" (UID: "b07d84bb-19ce-4d24-bee7-264d4b85ae22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:37:01.023249 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.023213    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:01.024718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.024695    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca" pod "b07d84bb-19ce-4d24-bee7-264d4b85ae22" (UID: "b07d84bb-19ce-4d24-bee7-264d4b85ae22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:37:01.024841 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.024754    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74" pod "b07d84bb-19ce-4d24-bee7-264d4b85ae22" (UID: "b07d84bb-19ce-4d24-bee7-264d4b85ae22"). InnerVolumeSpecName "kube-api-access-w9m74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:37:01.124585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.124559    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07d84bb-19ce-4d24-bee7-264d4b85ae22-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:01.124585 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.124585    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9m74\" (UniqueName: \"kubernetes.io/projected/b07d84bb-19ce-4d24-bee7-264d4b85ae22-kube-api-access-w9m74\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:01.749715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.749683    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"
Oct 30 03:37:01.749715 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.749700    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr" event={"ID":"b07d84bb-19ce-4d24-bee7-264d4b85ae22","Type":"ContainerDied","Data":"04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f"}
Oct 30 03:37:01.749929 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:01.749742    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04ec543da83e0bfa3b8bdd10a251068110ecad80177603eb0f9917faa4136d2f"
Oct 30 03:37:09.168314 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:37:09.168271    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:37:11.740452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:11.740376    4214 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xmv89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:37:11.741115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:11.741062    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xmv89" podUID="f4e682e6-c3a8-434f-9fae-06137c443148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:37:18.746152 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.746070    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"]
Oct 30 03:37:18.760333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.760301    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m6cpr"]
Oct 30 03:37:18.794260 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.794210    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"]
Oct 30 03:37:18.794689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.794675    4214 cpu_manager.go:414] "containerMap: removing container" podUID="b07d84bb-19ce-4d24-bee7-264d4b85ae22" containerName="revoke-keys"
Oct 30 03:37:18.794792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.794690    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07d84bb-19ce-4d24-bee7-264d4b85ae22" containerName="revoke-keys"
Oct 30 03:37:18.794792 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.794754    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07d84bb-19ce-4d24-bee7-264d4b85ae22" containerName="revoke-keys"
Oct 30 03:37:18.798021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.797996    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.814322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.814294    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:37:18.814514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.814399    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:37:18.817567 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.817540    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"]
Oct 30 03:37:18.833566 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.833529    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.833786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.833756    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpnn5\" (UniqueName: \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.833842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.833812    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.935275 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.935180    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.935449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.935352    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpnn5\" (UniqueName: \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.935449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.935377    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.935944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.935921    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.937460 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.937438    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:18.956090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:18.956065    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpnn5\" (UniqueName: \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\") pod \"maas-api-revoke-sub-3d4156f1-gngkc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:19.065297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.065256    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07d84bb-19ce-4d24-bee7-264d4b85ae22" path="/var/lib/kubelet/pods/b07d84bb-19ce-4d24-bee7-264d4b85ae22/volumes"
Oct 30 03:37:19.111802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.111774    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:19.410639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.410608    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"]
Oct 30 03:37:19.411522 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:37:19.411496    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729b2fc3_2f63_4226_8827_d78d69701cdc.slice/crio-45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82 WatchSource:0}: Error finding container 45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82: Status 404 returned error can't find the container with id 45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82
Oct 30 03:37:19.882499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.882462    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc" event={"ID":"729b2fc3-2f63-4226-8827-d78d69701cdc","Type":"ContainerStarted","Data":"b65e37e87a383a6bd4b5c557b9bbd8896e06c502a44ea94c34e465f0889f5ab5"}
Oct 30 03:37:19.882955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.882513    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc" event={"ID":"729b2fc3-2f63-4226-8827-d78d69701cdc","Type":"ContainerStarted","Data":"45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82"}
Oct 30 03:37:19.924148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:19.924039    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc" podStartSLOduration=1.924026826 podStartE2EDuration="1.924026826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:19.921203498 +0000 UTC m=+3378.995374291" watchObservedRunningTime="2026-10-30 03:37:19.924026826 +0000 UTC m=+3378.998197618"
Oct 30 03:37:20.889894 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:20.889862    4214 generic.go:356] "Generic (PLEG): container finished" podID="729b2fc3-2f63-4226-8827-d78d69701cdc" containerID="b65e37e87a383a6bd4b5c557b9bbd8896e06c502a44ea94c34e465f0889f5ab5" exitCode=0
Oct 30 03:37:20.890301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:20.889950    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc" event={"ID":"729b2fc3-2f63-4226-8827-d78d69701cdc","Type":"ContainerDied","Data":"b65e37e87a383a6bd4b5c557b9bbd8896e06c502a44ea94c34e465f0889f5ab5"}
Oct 30 03:37:22.042194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.042170    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:22.065393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.065369    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\") pod \"729b2fc3-2f63-4226-8827-d78d69701cdc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") "
Oct 30 03:37:22.065499 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.065457    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\" (UniqueName: \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\") pod \"729b2fc3-2f63-4226-8827-d78d69701cdc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") "
Oct 30 03:37:22.065562 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.065493    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\") pod \"729b2fc3-2f63-4226-8827-d78d69701cdc\" (UID: \"729b2fc3-2f63-4226-8827-d78d69701cdc\") "
Oct 30 03:37:22.065864 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.065843    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca" pod "729b2fc3-2f63-4226-8827-d78d69701cdc" (UID: "729b2fc3-2f63-4226-8827-d78d69701cdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:37:22.067373 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.067346    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca" pod "729b2fc3-2f63-4226-8827-d78d69701cdc" (UID: "729b2fc3-2f63-4226-8827-d78d69701cdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:37:22.067470 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.067452    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5" pod "729b2fc3-2f63-4226-8827-d78d69701cdc" (UID: "729b2fc3-2f63-4226-8827-d78d69701cdc"). InnerVolumeSpecName "kube-api-access-vpnn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:37:22.166591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.166515    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:22.166591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.166543    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpnn5\" (UniqueName: \"kubernetes.io/projected/729b2fc3-2f63-4226-8827-d78d69701cdc-kube-api-access-vpnn5\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:22.166591 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.166552    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729b2fc3-2f63-4226-8827-d78d69701cdc-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:22.903206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.903178    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"
Oct 30 03:37:22.903407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.903213    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc" event={"ID":"729b2fc3-2f63-4226-8827-d78d69701cdc","Type":"ContainerDied","Data":"45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82"}
Oct 30 03:37:22.903407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:22.903285    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e29e34b4d59311a078cfda8fb45a32e8978b123258be69c0ec58a75f71fe82"
Oct 30 03:37:36.214349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.214319    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"]
Oct 30 03:37:36.227254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.227204    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gngkc"]
Oct 30 03:37:36.261075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.261048    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"]
Oct 30 03:37:36.261589 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.261569    4214 cpu_manager.go:414] "containerMap: removing container" podUID="729b2fc3-2f63-4226-8827-d78d69701cdc" containerName="revoke-keys"
Oct 30 03:37:36.261694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.261592    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729b2fc3-2f63-4226-8827-d78d69701cdc" containerName="revoke-keys"
Oct 30 03:37:36.261764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.261727    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="729b2fc3-2f63-4226-8827-d78d69701cdc" containerName="revoke-keys"
Oct 30 03:37:36.265222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.265201    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.282758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.282715    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:37:36.285634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.285612    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:37:36.287564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.287534    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"]
Oct 30 03:37:36.407736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.407701    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.407891 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.407754    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.407933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.407890    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnf4\" (UniqueName: \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.508950 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.508876    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnf4\" (UniqueName: \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.509091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.508954    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.509091 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.509002    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.509711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.509691    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.511290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.511266    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.528419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.528393    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnf4\" (UniqueName: \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\") pod \"maas-api-revoke-sub-3d4156f1-cgk4l\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.579915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.579887    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:36.882323 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.882295    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"]
Oct 30 03:37:36.884658 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:37:36.884626    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5303e8a9_b387_40aa_a8eb_e9e828486ae3.slice/crio-b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c WatchSource:0}: Error finding container b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c: Status 404 returned error can't find the container with id b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c
Oct 30 03:37:36.998474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:36.998436    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l" event={"ID":"5303e8a9-b387-40aa-a8eb-e9e828486ae3","Type":"ContainerStarted","Data":"b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c"}
Oct 30 03:37:37.030017 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:37.029985    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"]
Oct 30 03:37:37.035849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:37.035825    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cbcc9b7-xgvtb"]
Oct 30 03:37:37.065242 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:37.065206    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff0bb34-f3e0-4a2a-a269-0ca716ee1589" path="/var/lib/kubelet/pods/2ff0bb34-f3e0-4a2a-a269-0ca716ee1589/volumes"
Oct 30 03:37:37.065683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:37.065667    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729b2fc3-2f63-4226-8827-d78d69701cdc" path="/var/lib/kubelet/pods/729b2fc3-2f63-4226-8827-d78d69701cdc/volumes"
Oct 30 03:37:38.006474 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:38.006377    4214 generic.go:356] "Generic (PLEG): container finished" podID="5303e8a9-b387-40aa-a8eb-e9e828486ae3" containerID="7398d690a165d7dbc495c9de4c44e2fb3b842519ee6403c5ac65fde0d55e4fab" exitCode=0
Oct 30 03:37:38.006834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:38.006468    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l" event={"ID":"5303e8a9-b387-40aa-a8eb-e9e828486ae3","Type":"ContainerDied","Data":"7398d690a165d7dbc495c9de4c44e2fb3b842519ee6403c5ac65fde0d55e4fab"}
Oct 30 03:37:39.147071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.147042    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:39.162994 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:37:39.162959    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:37:39.234886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.234853    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\") pod \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") "
Oct 30 03:37:39.235062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.234899    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\") pod \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") "
Oct 30 03:37:39.235062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.235000    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\" (UniqueName: \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\") pod \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\" (UID: \"5303e8a9-b387-40aa-a8eb-e9e828486ae3\") "
Oct 30 03:37:39.235333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.235308    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca" pod "5303e8a9-b387-40aa-a8eb-e9e828486ae3" (UID: "5303e8a9-b387-40aa-a8eb-e9e828486ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:37:39.235419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.235406    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:39.236980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.236947    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca" pod "5303e8a9-b387-40aa-a8eb-e9e828486ae3" (UID: "5303e8a9-b387-40aa-a8eb-e9e828486ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:37:39.237086 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.237012    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4" pod "5303e8a9-b387-40aa-a8eb-e9e828486ae3" (UID: "5303e8a9-b387-40aa-a8eb-e9e828486ae3"). InnerVolumeSpecName "kube-api-access-5qnf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:37:39.336439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.336402    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5303e8a9-b387-40aa-a8eb-e9e828486ae3-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:39.336439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:39.336436    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnf4\" (UniqueName: \"kubernetes.io/projected/5303e8a9-b387-40aa-a8eb-e9e828486ae3-kube-api-access-5qnf4\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:37:40.020614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:40.020581    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"
Oct 30 03:37:40.020797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:40.020615    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l" event={"ID":"5303e8a9-b387-40aa-a8eb-e9e828486ae3","Type":"ContainerDied","Data":"b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c"}
Oct 30 03:37:40.020797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:40.020663    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b300122175db394e83f1e952ca815d7b4ee7b0edbba65f7acf94828bbf028c3c"
Oct 30 03:37:52.034545 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.034504    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:37:52.035385 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.035361    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5303e8a9-b387-40aa-a8eb-e9e828486ae3" containerName="revoke-keys"
Oct 30 03:37:52.035468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.035387    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5303e8a9-b387-40aa-a8eb-e9e828486ae3" containerName="revoke-keys"
Oct 30 03:37:52.035527 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.035507    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5303e8a9-b387-40aa-a8eb-e9e828486ae3" containerName="revoke-keys"
Oct 30 03:37:52.039638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.039616    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.055586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.055556    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:37:52.063682 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.063657    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ed4e18-openshift-default-dockercfg-6mcvq\""
Oct 30 03:37:52.165873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.165719    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.165873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.165785    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.165873 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.165814    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.165955    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.166000    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.166084    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbmv\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.166152    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.166190    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.166530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.166270    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267177    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbmv\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267215 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267281    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267313    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267347    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267402    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267427    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267446    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267472    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.267595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267489    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.268048 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.267993    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.268201 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.268120    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.268317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.268221    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.268317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.268294    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.268554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.268533    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.269828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.269806    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.270534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.270514    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.299884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.297594    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbmv\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.299884 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.299777    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\") pod \"e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") " pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.358475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.358428    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:52.678816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.678317    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:37:52.692608 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.692560    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:37:52.692721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.692657    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:37:52.692721 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:52.692698    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:37:53.101002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:53.100969    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerStarted","Data":"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"}
Oct 30 03:37:53.101502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:53.101031    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerStarted","Data":"6deab6b0dc5e79174b7014ee16a579e57cabf569713707b5198cc73c0802d614"}
Oct 30 03:37:53.143793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:53.143742    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podStartSLOduration=2.143727453 podStartE2EDuration="2.143727453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:53.142981821 +0000 UTC m=+3412.217152615" watchObservedRunningTime="2026-10-30 03:37:53.143727453 +0000 UTC m=+3412.217898245"
Oct 30 03:37:53.358820 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:53.358726    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:53.364520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:53.364489    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:54.107615 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:54.107582    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:54.108506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:54.108487    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:37:57.537453 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.537415    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:37:57.543883 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.543854    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.565806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.565776    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:37:57.576344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.576315    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ed4e18\""
Oct 30 03:37:57.576750 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.576725    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ffe9d7be-dockercfg-gwcqx\""
Oct 30 03:37:57.626105 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.626072    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.626315 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.626131    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp6d\" (UniqueName: \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.726646 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.726607    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.726824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.726673    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp6d\" (UniqueName: \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.727352 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.727330    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.746408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.746382    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp6d\" (UniqueName: \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\") pod \"payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:57.882880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:57.882844    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:58.183637 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:58.183605    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:37:58.185351 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:37:58.185314    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcc386e_0731_4c00_9b11_5dda874a1f15.slice/crio-d17bcab6d32b45c454f5907fed656188dd4ea20b4cb13547baaf4eb13beb6865 WatchSource:0}: Error finding container d17bcab6d32b45c454f5907fed656188dd4ea20b4cb13547baaf4eb13beb6865: Status 404 returned error can't find the container with id d17bcab6d32b45c454f5907fed656188dd4ea20b4cb13547baaf4eb13beb6865
Oct 30 03:37:59.142694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:59.142631    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" event={"ID":"dfcc386e-0731-4c00-9b11-5dda874a1f15","Type":"ContainerStarted","Data":"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"}
Oct 30 03:37:59.143194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:59.142713    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" event={"ID":"dfcc386e-0731-4c00-9b11-5dda874a1f15","Type":"ContainerStarted","Data":"d17bcab6d32b45c454f5907fed656188dd4ea20b4cb13547baaf4eb13beb6865"}
Oct 30 03:37:59.143194 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:59.142862    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:37:59.184483 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:37:59.184429    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" podStartSLOduration=2.184414368 podStartE2EDuration="2.184414368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:59.18233986 +0000 UTC m=+3418.256510670" watchObservedRunningTime="2026-10-30 03:37:59.184414368 +0000 UTC m=+3418.258585161"
Oct 30 03:38:09.165346 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:38:09.165308    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:38:09.962573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:09.961266    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": read tcp 10.133.0.2:43490->10.133.1.61:15021: read: connection reset by peer" start-of-body=
Oct 30 03:38:09.962573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:09.961337    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": read tcp 10.133.0.2:43490->10.133.1.61:15021: read: connection reset by peer"
Oct 30 03:38:10.151435 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.151360    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:38:10.238951 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.238865    4214 generic.go:356] "Generic (PLEG): container finished" podID="acd1f590-c823-48f2-a730-2374a0c955f8" containerID="108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13" exitCode=0
Oct 30 03:38:10.239388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.238945    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerDied","Data":"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"}
Oct 30 03:38:10.239388 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.239346    4214 scope.go:122] "RemoveContainer" containerID="108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"
Oct 30 03:38:10.240552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.240521    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:38:10.240624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.240596    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:38:10.240663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:10.240632    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:38:11.265670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:11.265627    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerStarted","Data":"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"}
Oct 30 03:38:11.359688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:11.359652    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:38:11.361220 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:11.361188    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 03:38:11.361502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:11.361473    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 03:38:12.362254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:12.361759    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 03:38:12.362254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:12.361828    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 03:38:13.359714 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:13.359675    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 03:38:13.359945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:13.359747    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 03:38:14.363254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:14.362659    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 03:38:14.363254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:14.362723    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 03:38:15.362735 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:15.362696    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body=
Oct 30 03:38:15.363066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:15.363034    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused"
Oct 30 03:38:16.363341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:16.363310    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:38:16.363835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:16.363705    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:38:16.364496 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:16.364476    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:38:24.028874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:24.028837    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"]
Oct 30 03:38:24.041489 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:24.041460    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mgps4"]
Oct 30 03:38:25.065354 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:25.065311    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de56eba-3ad6-4a03-9d1f-847450af80a8" path="/var/lib/kubelet/pods/6de56eba-3ad6-4a03-9d1f-847450af80a8/volumes"
Oct 30 03:38:36.398509 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.398477    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"]
Oct 30 03:38:36.402247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.402218    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.421121 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.421094    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"]
Oct 30 03:38:36.430265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.430243    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:38:36.430409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.430358    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:38:36.557082 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.557055    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7q7\" (UniqueName: \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.557223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.557132    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.557223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.557196    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.658316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.658205    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.658469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.658371    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.658469 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.658444    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7q7\" (UniqueName: \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.659112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.659087    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.660778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.660738    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.680516 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.680484    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7q7\" (UniqueName: \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\") pod \"maas-api-revoke-sub-f7957080-5fjhv\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:36.715089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:36.715068    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:37.023107 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:37.023079    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"]
Oct 30 03:38:37.023779 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:38:37.023749    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7679e79_59c8_4169_a056_d48b087ab1b5.slice/crio-04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f WatchSource:0}: Error finding container 04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f: Status 404 returned error can't find the container with id 04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f
Oct 30 03:38:37.442553 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:37.442518    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv" event={"ID":"f7679e79-59c8-4169-a056-d48b087ab1b5","Type":"ContainerStarted","Data":"79404c58ef3079efea18bedeb65e84da70691704b3415611b75c04b6bc24a646"}
Oct 30 03:38:37.442976 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:37.442586    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv" event={"ID":"f7679e79-59c8-4169-a056-d48b087ab1b5","Type":"ContainerStarted","Data":"04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f"}
Oct 30 03:38:37.484068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:37.484022    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv" podStartSLOduration=1.484008911 podStartE2EDuration="1.484008911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:37.482105498 +0000 UTC m=+3456.556276296" watchObservedRunningTime="2026-10-30 03:38:37.484008911 +0000 UTC m=+3456.558179704"
Oct 30 03:38:38.450776 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:38.450738    4214 generic.go:356] "Generic (PLEG): container finished" podID="f7679e79-59c8-4169-a056-d48b087ab1b5" containerID="79404c58ef3079efea18bedeb65e84da70691704b3415611b75c04b6bc24a646" exitCode=0
Oct 30 03:38:38.451172 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:38.450813    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv" event={"ID":"f7679e79-59c8-4169-a056-d48b087ab1b5","Type":"ContainerDied","Data":"79404c58ef3079efea18bedeb65e84da70691704b3415611b75c04b6bc24a646"}
Oct 30 03:38:39.163090 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:38:39.163053    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:38:39.591355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.591330    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:39.684112 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.684078    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\") pod \"f7679e79-59c8-4169-a056-d48b087ab1b5\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") "
Oct 30 03:38:39.684322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.684145    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\") pod \"f7679e79-59c8-4169-a056-d48b087ab1b5\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") "
Oct 30 03:38:39.684322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.684221    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\" (UniqueName: \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\") pod \"f7679e79-59c8-4169-a056-d48b087ab1b5\" (UID: \"f7679e79-59c8-4169-a056-d48b087ab1b5\") "
Oct 30 03:38:39.684538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.684517    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca" pod "f7679e79-59c8-4169-a056-d48b087ab1b5" (UID: "f7679e79-59c8-4169-a056-d48b087ab1b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:39.684658 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.684640    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:39.686277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.686242    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca" pod "f7679e79-59c8-4169-a056-d48b087ab1b5" (UID: "f7679e79-59c8-4169-a056-d48b087ab1b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:39.686396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.686359    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7" pod "f7679e79-59c8-4169-a056-d48b087ab1b5" (UID: "f7679e79-59c8-4169-a056-d48b087ab1b5"). InnerVolumeSpecName "kube-api-access-8t7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:39.785932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.785860    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7q7\" (UniqueName: \"kubernetes.io/projected/f7679e79-59c8-4169-a056-d48b087ab1b5-kube-api-access-8t7q7\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:39.785932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:39.785893    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7679e79-59c8-4169-a056-d48b087ab1b5-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:40.464217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:40.464176    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv" event={"ID":"f7679e79-59c8-4169-a056-d48b087ab1b5","Type":"ContainerDied","Data":"04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f"}
Oct 30 03:38:40.464217 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:40.464196    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"
Oct 30 03:38:40.464461 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:40.464255    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c5d1cd890108e10f2aa706202caa4a8df3b0f680acbbca5e71597381c8739f"
Oct 30 03:38:47.652429 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.652333    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"]
Oct 30 03:38:47.652915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.652785    4214 cpu_manager.go:414] "containerMap: removing container" podUID="f7679e79-59c8-4169-a056-d48b087ab1b5" containerName="revoke-keys"
Oct 30 03:38:47.652915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.652798    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7679e79-59c8-4169-a056-d48b087ab1b5" containerName="revoke-keys"
Oct 30 03:38:47.652915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.652877    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7679e79-59c8-4169-a056-d48b087ab1b5" containerName="revoke-keys"
Oct 30 03:38:47.667912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.667878    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.680043 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.677603    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"]
Oct 30 03:38:47.683880 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.683856    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:38:47.684074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.684062    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:38:47.765393 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.765357    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdncf\" (UniqueName: \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.765552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.765517    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.765659 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.765639    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.866189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.866150    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.866377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.866207    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.866377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.866240    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdncf\" (UniqueName: \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.866955 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.866923    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.868603 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.868582    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.887014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.886983    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdncf\" (UniqueName: \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:47.992521 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:47.992441    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:48.301775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:48.301749    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"]
Oct 30 03:38:48.303658 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:38:48.303632    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca8a052_2be7_4701_acff_b5ce71eee570.slice/crio-29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3 WatchSource:0}: Error finding container 29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3: Status 404 returned error can't find the container with id 29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3
Oct 30 03:38:48.512689 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:48.512658    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x" event={"ID":"9ca8a052-2be7-4701-acff-b5ce71eee570","Type":"ContainerStarted","Data":"29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3"}
Oct 30 03:38:49.520259 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:49.520152    4214 generic.go:356] "Generic (PLEG): container finished" podID="9ca8a052-2be7-4701-acff-b5ce71eee570" containerID="720d8bd785e1d1f47ae4c46d516b836468b1314e83d8f7171a5ab4035cf69dc1" exitCode=0
Oct 30 03:38:49.520628 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:49.520252    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x" event={"ID":"9ca8a052-2be7-4701-acff-b5ce71eee570","Type":"ContainerDied","Data":"720d8bd785e1d1f47ae4c46d516b836468b1314e83d8f7171a5ab4035cf69dc1"}
Oct 30 03:38:50.680844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.680818    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:50.797162 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.797130    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\") pod \"9ca8a052-2be7-4701-acff-b5ce71eee570\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") "
Oct 30 03:38:50.797376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.797258    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\" (UniqueName: \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\") pod \"9ca8a052-2be7-4701-acff-b5ce71eee570\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") "
Oct 30 03:38:50.797376 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.797358    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\") pod \"9ca8a052-2be7-4701-acff-b5ce71eee570\" (UID: \"9ca8a052-2be7-4701-acff-b5ce71eee570\") "
Oct 30 03:38:50.797786 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.797754    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca" pod "9ca8a052-2be7-4701-acff-b5ce71eee570" (UID: "9ca8a052-2be7-4701-acff-b5ce71eee570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:50.799335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.799309    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf" pod "9ca8a052-2be7-4701-acff-b5ce71eee570" (UID: "9ca8a052-2be7-4701-acff-b5ce71eee570"). InnerVolumeSpecName "kube-api-access-fdncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:50.800330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.800305    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca" pod "9ca8a052-2be7-4701-acff-b5ce71eee570" (UID: "9ca8a052-2be7-4701-acff-b5ce71eee570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:50.898174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.898141    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdncf\" (UniqueName: \"kubernetes.io/projected/9ca8a052-2be7-4701-acff-b5ce71eee570-kube-api-access-fdncf\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:50.898174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.898168    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:50.898174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:50.898178    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca8a052-2be7-4701-acff-b5ce71eee570-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:38:51.535744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:51.535665    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"
Oct 30 03:38:51.535744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:51.535682    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x" event={"ID":"9ca8a052-2be7-4701-acff-b5ce71eee570","Type":"ContainerDied","Data":"29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3"}
Oct 30 03:38:51.535744 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:51.535735    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29cbb51eed84dbf9fadb2a2f47d8f0c146c06b709b1e2988c262bab775ec6dd3"
Oct 30 03:38:58.083468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:38:58.083434    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:39:00.502425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:00.502392    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"]
Oct 30 03:39:00.520611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:00.519836    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ed4e18-4fw8x"]
Oct 30 03:39:01.065570 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:01.065536    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca8a052-2be7-4701-acff-b5ce71eee570" path="/var/lib/kubelet/pods/9ca8a052-2be7-4701-acff-b5ce71eee570/volumes"
Oct 30 03:39:03.084529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:03.084492    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing" containerID="cri-o://35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa" gracePeriod=25
Oct 30 03:39:07.111270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:07.107083    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:39:07.111270 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:07.107352    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy" containerID="cri-o://0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85" gracePeriod=30
Oct 30 03:39:07.722993 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:07.722953    4214 scope.go:122] "RemoveContainer" containerID="1cb308fa64b589de50db0e1a9e346c4b3d6ee7e5645b6836d4d096cbe1460f97"
Oct 30 03:39:07.732674 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:07.732655    4214 scope.go:122] "RemoveContainer" containerID="1c200b9801e602eaf75db445955afb23dce4304775a67994ec39b5b9b330cf5b"
Oct 30 03:39:07.741529 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:07.741509    4214 scope.go:122] "RemoveContainer" containerID="7c03e3dcfc300dd12e3eb1bfb29ab4085af3dcb18677cf0fa3a72e7e02003aa7"
Oct 30 03:39:08.396213 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.396180    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"]
Oct 30 03:39:08.408609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.408583    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgk4l"]
Oct 30 03:39:08.439523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.439497    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"]
Oct 30 03:39:08.439920 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.439907    4214 cpu_manager.go:414] "containerMap: removing container" podUID="9ca8a052-2be7-4701-acff-b5ce71eee570" containerName="revoke-keys"
Oct 30 03:39:08.439966 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.439922    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca8a052-2be7-4701-acff-b5ce71eee570" containerName="revoke-keys"
Oct 30 03:39:08.440014 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.440005    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca8a052-2be7-4701-acff-b5ce71eee570" containerName="revoke-keys"
Oct 30 03:39:08.444613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.444598    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.460620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.460598    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"]
Oct 30 03:39:08.461551 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.461530    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:39:08.461675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.461654    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:39:08.587277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.587213    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.587452 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.587359    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.587502 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.587485    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vjv\" (UniqueName: \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.688316 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.688219    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.688486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.688394    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vjv\" (UniqueName: \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.688486 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.688446    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.689211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.689180    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.690626 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.690600    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.708719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.708689    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vjv\" (UniqueName: \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\") pod \"maas-api-revoke-sub-3d4156f1-6nbdj\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:08.757639 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:08.757613    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:09.061809 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:39:09.061778    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cfd86c_c5a8_458c_917d_d329b6233fab.slice/crio-154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b WatchSource:0}: Error finding container 154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b: Status 404 returned error can't find the container with id 154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b
Oct 30 03:39:09.066835 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:09.066810    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5303e8a9-b387-40aa-a8eb-e9e828486ae3" path="/var/lib/kubelet/pods/5303e8a9-b387-40aa-a8eb-e9e828486ae3/volumes"
Oct 30 03:39:09.067134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:09.067120    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"]
Oct 30 03:39:09.167780 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:09.167732    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:39:09.644007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:09.643916    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj" event={"ID":"63cfd86c-c5a8-458c-917d-d329b6233fab","Type":"ContainerStarted","Data":"7151b5c3cc6cbfbe6e01b7cb39bd211d0590b9cf590622581d541bc814b5405a"}
Oct 30 03:39:09.644007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:09.643997    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj" event={"ID":"63cfd86c-c5a8-458c-917d-d329b6233fab","Type":"ContainerStarted","Data":"154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b"}
Oct 30 03:39:09.685346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:09.685297    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj" podStartSLOduration=1.685283218 podStartE2EDuration="1.685283218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:09.684146432 +0000 UTC m=+3488.758317215" watchObservedRunningTime="2026-10-30 03:39:09.685283218 +0000 UTC m=+3488.759454022"
Oct 30 03:39:10.150021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:10.149979    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:39:10.652136 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:10.652096    4214 generic.go:356] "Generic (PLEG): container finished" podID="63cfd86c-c5a8-458c-917d-d329b6233fab" containerID="7151b5c3cc6cbfbe6e01b7cb39bd211d0590b9cf590622581d541bc814b5405a" exitCode=0
Oct 30 03:39:10.652663 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:10.652149    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj" event={"ID":"63cfd86c-c5a8-458c-917d-d329b6233fab","Type":"ContainerDied","Data":"7151b5c3cc6cbfbe6e01b7cb39bd211d0590b9cf590622581d541bc814b5405a"}
Oct 30 03:39:11.794718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.794696    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:11.922959 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.922885    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\") pod \"63cfd86c-c5a8-458c-917d-d329b6233fab\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") "
Oct 30 03:39:11.923114 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.922963    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\") pod \"63cfd86c-c5a8-458c-917d-d329b6233fab\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") "
Oct 30 03:39:11.923159 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.923134    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\" (UniqueName: \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\") pod \"63cfd86c-c5a8-458c-917d-d329b6233fab\" (UID: \"63cfd86c-c5a8-458c-917d-d329b6233fab\") "
Oct 30 03:39:11.923384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.923355    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca" pod "63cfd86c-c5a8-458c-917d-d329b6233fab" (UID: "63cfd86c-c5a8-458c-917d-d329b6233fab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:11.924970 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.924943    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca" pod "63cfd86c-c5a8-458c-917d-d329b6233fab" (UID: "63cfd86c-c5a8-458c-917d-d329b6233fab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:11.925116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:11.925100    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv" pod "63cfd86c-c5a8-458c-917d-d329b6233fab" (UID: "63cfd86c-c5a8-458c-917d-d329b6233fab"). InnerVolumeSpecName "kube-api-access-p6vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:12.024980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.024952    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.024980 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.024978    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cfd86c-c5a8-458c-917d-d329b6233fab-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.025151 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.024989    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vjv\" (UniqueName: \"kubernetes.io/projected/63cfd86c-c5a8-458c-917d-d329b6233fab-kube-api-access-p6vjv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.356669 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.356643    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:39:12.529724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529643    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.529724 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529717    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.529964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529743    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.529964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529782    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.529964 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529916    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.530143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.529981    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.530143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530027    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.530143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530058    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.530143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530074    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:12.530143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530131    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\") pod \"acd1f590-c823-48f2-a730-2374a0c955f8\" (UID: \"acd1f590-c823-48f2-a730-2374a0c955f8\") "
Oct 30 03:39:12.530449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530293    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:12.530449 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530303    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:12.530572 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530553    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:12.530636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530576    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd1f590-c823-48f2-a730-2374a0c955f8-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.530636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530589    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:12.530636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530596    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.530764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.530637    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.532053 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.532034    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:12.532143 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.532061    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "kube-api-access-vnbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:12.532383 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.532363    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:39:12.532425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.532369    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy" pod "acd1f590-c823-48f2-a730-2374a0c955f8" (UID: "acd1f590-c823-48f2-a730-2374a0c955f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:12.631097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631068    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.631097 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631093    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbmv\" (UniqueName: \"kubernetes.io/projected/acd1f590-c823-48f2-a730-2374a0c955f8-kube-api-access-vnbmv\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.631290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631104    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.631290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631113    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.631290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631121    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd1f590-c823-48f2-a730-2374a0c955f8-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.631290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.631129    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd1f590-c823-48f2-a730-2374a0c955f8-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:12.664660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.664638    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"
Oct 30 03:39:12.664785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.664670    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj" event={"ID":"63cfd86c-c5a8-458c-917d-d329b6233fab","Type":"ContainerDied","Data":"154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b"}
Oct 30 03:39:12.664785 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.664721    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154b7af106317491332f5e4111f9382a8c3c932900859581f13de5e19aa35b1b"
Oct 30 03:39:12.666560 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.666536    4214 generic.go:356] "Generic (PLEG): container finished" podID="acd1f590-c823-48f2-a730-2374a0c955f8" containerID="0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85" exitCode=0
Oct 30 03:39:12.666676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.666575    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"
Oct 30 03:39:12.666676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.666615    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerDied","Data":"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"}
Oct 30 03:39:12.666676 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.666669    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l" event={"ID":"acd1f590-c823-48f2-a730-2374a0c955f8","Type":"ContainerDied","Data":"6deab6b0dc5e79174b7014ee16a579e57cabf569713707b5198cc73c0802d614"}
Oct 30 03:39:12.666858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.666698    4214 scope.go:122] "RemoveContainer" containerID="0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"
Oct 30 03:39:12.677127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.677112    4214 scope.go:122] "RemoveContainer" containerID="108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"
Oct 30 03:39:12.685380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.685365    4214 scope.go:122] "RemoveContainer" containerID="0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"
Oct 30 03:39:12.685655 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:12.685640    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85\": container with ID starting with 0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85 not found: ID does not exist" containerID="0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"
Oct 30 03:39:12.685700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.685664    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85"} err="failed to get container status \"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85\": rpc error: code = NotFound desc = could not find container \"0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85\": container with ID starting with 0bfb81f39efde6ef77e79d40682ecf0feac2bd5a1b528d2d0a0e267247449d85 not found: ID does not exist"
Oct 30 03:39:12.685700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.685678    4214 scope.go:122] "RemoveContainer" containerID="108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"
Oct 30 03:39:12.685891 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:12.685874    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13\": container with ID starting with 108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13 not found: ID does not exist" containerID="108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"
Oct 30 03:39:12.685925 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.685896    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13"} err="failed to get container status \"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13\": rpc error: code = NotFound desc = could not find container \"108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13\": container with ID starting with 108f9671504ac035863431aee28bc6858a5d92e02aa8673b9620c06b395baa13 not found: ID does not exist"
Oct 30 03:39:12.720076 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.720044    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:39:12.736198 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:12.736128    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ed4e18-openshift-default-6d88cfdc4f2vl4l"]
Oct 30 03:39:13.065494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:13.065459    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" path="/var/lib/kubelet/pods/acd1f590-c823-48f2-a730-2374a0c955f8/volumes"
Oct 30 03:39:15.027842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:15.027809    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"]
Oct 30 03:39:15.041028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:15.040991    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm82z"]
Oct 30 03:39:15.066522 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:15.066494    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fc6fe4-3676-42d5-834f-688b6cf2365c" path="/var/lib/kubelet/pods/57fc6fe4-3676-42d5-834f-688b6cf2365c/volumes"
Oct 30 03:39:20.150180 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:20.150130    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:39:23.338787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.338764    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:39:23.440208 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.440133    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\") pod \"dfcc386e-0731-4c00-9b11-5dda874a1f15\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") "
Oct 30 03:39:23.440384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.440284    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\" (UniqueName: \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\") pod \"dfcc386e-0731-4c00-9b11-5dda874a1f15\" (UID: \"dfcc386e-0731-4c00-9b11-5dda874a1f15\") "
Oct 30 03:39:23.440532 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.440510    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume" pod "dfcc386e-0731-4c00-9b11-5dda874a1f15" (UID: "dfcc386e-0731-4c00-9b11-5dda874a1f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:23.442333 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.442303    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d" pod "dfcc386e-0731-4c00-9b11-5dda874a1f15" (UID: "dfcc386e-0731-4c00-9b11-5dda874a1f15"). InnerVolumeSpecName "kube-api-access-sfp6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:23.541518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.541482    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfp6d\" (UniqueName: \"kubernetes.io/projected/dfcc386e-0731-4c00-9b11-5dda874a1f15-kube-api-access-sfp6d\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.541518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.541512    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc386e-0731-4c00-9b11-5dda874a1f15-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.734806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.734724    4214 generic.go:356] "Generic (PLEG): container finished" podID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerID="35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa" exitCode=0
Oct 30 03:39:23.734806 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.734768    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"
Oct 30 03:39:23.735007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.734812    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" event={"ID":"dfcc386e-0731-4c00-9b11-5dda874a1f15","Type":"ContainerDied","Data":"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"}
Oct 30 03:39:23.735007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.734866    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96" event={"ID":"dfcc386e-0731-4c00-9b11-5dda874a1f15","Type":"ContainerDied","Data":"d17bcab6d32b45c454f5907fed656188dd4ea20b4cb13547baaf4eb13beb6865"}
Oct 30 03:39:23.735007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.734889    4214 scope.go:122] "RemoveContainer" containerID="35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"
Oct 30 03:39:23.745038 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.745015    4214 scope.go:122] "RemoveContainer" containerID="35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"
Oct 30 03:39:23.745312 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:23.745290    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa\": container with ID starting with 35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa not found: ID does not exist" containerID="35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"
Oct 30 03:39:23.745423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.745319    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa"} err="failed to get container status \"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa\": rpc error: code = NotFound desc = could not find container \"35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa\": container with ID starting with 35e382fa9e9c80335ddd1386ba6c75aee3b94bd0ccd4270f5eaecd6e583706aa not found: ID does not exist"
Oct 30 03:39:23.788915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.788880    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:39:23.792326 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:23.792294    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ed4e18-78d5856c6xkw96"]
Oct 30 03:39:25.064787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:25.064751    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" path="/var/lib/kubelet/pods/dfcc386e-0731-4c00-9b11-5dda874a1f15/volumes"
Oct 30 03:39:26.270170 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.270131    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 03:39:26.270633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.270355    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" podUID="6627dcdc-059c-41ba-b173-f91031ad4db1" containerName="manager" containerID="cri-o://04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4" gracePeriod=10
Oct 30 03:39:26.530207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.530145    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 03:39:26.572018 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.571996    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\" (UniqueName: \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\") pod \"6627dcdc-059c-41ba-b173-f91031ad4db1\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") "
Oct 30 03:39:26.572181 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.572030    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\") pod \"6627dcdc-059c-41ba-b173-f91031ad4db1\" (UID: \"6627dcdc-059c-41ba-b173-f91031ad4db1\") "
Oct 30 03:39:26.572586 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.572560    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume" pod "6627dcdc-059c-41ba-b173-f91031ad4db1" (UID: "6627dcdc-059c-41ba-b173-f91031ad4db1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:26.574753 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.574707    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td" pod "6627dcdc-059c-41ba-b173-f91031ad4db1" (UID: "6627dcdc-059c-41ba-b173-f91031ad4db1"). InnerVolumeSpecName "kube-api-access-mr4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:26.673340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.673308    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr4td\" (UniqueName: \"kubernetes.io/projected/6627dcdc-059c-41ba-b173-f91031ad4db1-kube-api-access-mr4td\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:26.673340 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.673337    4214 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627dcdc-059c-41ba-b173-f91031ad4db1-extensions-socket-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:39:26.754074 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.754041    4214 generic.go:356] "Generic (PLEG): container finished" podID="6627dcdc-059c-41ba-b173-f91031ad4db1" containerID="04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4" exitCode=0
Oct 30 03:39:26.754247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.754094    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"
Oct 30 03:39:26.754247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.754132    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" event={"ID":"6627dcdc-059c-41ba-b173-f91031ad4db1","Type":"ContainerDied","Data":"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"}
Oct 30 03:39:26.754247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.754182    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7" event={"ID":"6627dcdc-059c-41ba-b173-f91031ad4db1","Type":"ContainerDied","Data":"4a739e459ed354cfdca54cd4d0d96c1c7e1e4a8a61ce6cf3ab11949f1a66d41e"}
Oct 30 03:39:26.754247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.754203    4214 scope.go:122] "RemoveContainer" containerID="04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"
Oct 30 03:39:26.764531 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.764515    4214 scope.go:122] "RemoveContainer" containerID="04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"
Oct 30 03:39:26.764794 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:26.764779    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4\": container with ID starting with 04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4 not found: ID does not exist" containerID="04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"
Oct 30 03:39:26.764842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.764803    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4"} err="failed to get container status \"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4\": rpc error: code = NotFound desc = could not find container \"04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4\": container with ID starting with 04eb7bf2c6036c8e53f7b3f651c12815a408de5c81a6b53ad0b87d92a12df7f4 not found: ID does not exist"
Oct 30 03:39:26.812378 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.812336    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 03:39:26.819815 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:26.819789    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-44dk7"]
Oct 30 03:39:27.065319 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:27.065244    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6627dcdc-059c-41ba-b173-f91031ad4db1" path="/var/lib/kubelet/pods/6627dcdc-059c-41ba-b173-f91031ad4db1/volumes"
Oct 30 03:39:35.027919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:35.027885    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"]
Oct 30 03:39:35.040800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:35.040761    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8vjx7"]
Oct 30 03:39:35.065281 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:35.065247    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa57a4cd-6c66-4933-a210-f7e840452ab4" path="/var/lib/kubelet/pods/aa57a4cd-6c66-4933-a210-f7e840452ab4/volumes"
Oct 30 03:39:39.163077 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:39:39.163022    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:39:46.027849 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:46.027811    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"]
Oct 30 03:39:46.039942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:46.039915    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-p97ql"]
Oct 30 03:39:47.065526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:39:47.065487    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fbaa96-ff8c-42f6-9014-4782a8ca04ed" path="/var/lib/kubelet/pods/b0fbaa96-ff8c-42f6-9014-4782a8ca04ed/volumes"
Oct 30 03:40:07.785539 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:07.785505    4214 scope.go:122] "RemoveContainer" containerID="c69094fbc3687041d00afa9d5bb37214de507e401e15954897d34cc8b91100bd"
Oct 30 03:40:09.163623 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:40:09.163586    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:40:15.030448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:15.030355    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"]
Oct 30 03:40:15.043108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:15.043082    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-722c4"]
Oct 30 03:40:15.064406 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:15.064376    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d42ddd-3dd0-4c9a-93fd-8c8644afcbed" path="/var/lib/kubelet/pods/14d42ddd-3dd0-4c9a-93fd-8c8644afcbed/volumes"
Oct 30 03:40:25.031210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:25.031178    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"]
Oct 30 03:40:25.043901 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:25.043875    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lkltj"]
Oct 30 03:40:25.065337 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:25.065309    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa452646-8139-47a4-89ae-153a7b7eee1b" path="/var/lib/kubelet/pods/fa452646-8139-47a4-89ae-153a7b7eee1b/volumes"
Oct 30 03:40:29.417796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.417756    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"]
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418192    4214 cpu_manager.go:414] "containerMap: removing container" podUID="6627dcdc-059c-41ba-b173-f91031ad4db1" containerName="manager"
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418206    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6627dcdc-059c-41ba-b173-f91031ad4db1" containerName="manager"
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418223    4214 cpu_manager.go:414] "containerMap: removing container" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing"
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418251    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing"
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418265    4214 cpu_manager.go:414] "containerMap: removing container" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.418276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418274    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418286    4214 cpu_manager.go:414] "containerMap: removing container" podUID="63cfd86c-c5a8-458c-917d-d329b6233fab" containerName="revoke-keys"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418291    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cfd86c-c5a8-458c-917d-d329b6233fab" containerName="revoke-keys"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418302    4214 cpu_manager.go:414] "containerMap: removing container" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418307    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418389    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cfd86c-c5a8-458c-917d-d329b6233fab" containerName="revoke-keys"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418399    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418406    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="6627dcdc-059c-41ba-b173-f91031ad4db1" containerName="manager"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418413    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcc386e-0731-4c00-9b11-5dda874a1f15" containerName="payload-processing"
Oct 30 03:40:29.418537 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.418418    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd1f590-c823-48f2-a730-2374a0c955f8" containerName="istio-proxy"
Oct 30 03:40:29.421782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.421764    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.436684 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.436659    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"]
Oct 30 03:40:29.446722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.446700    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g7ggq\""
Oct 30 03:40:29.576506 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.576462    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvbw\" (UniqueName: \"kubernetes.io/projected/2b082597-6fd3-484a-9e6b-d9d400d0921a-kube-api-access-plvbw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.576691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.576611    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b082597-6fd3-484a-9e6b-d9d400d0921a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.677826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.677741    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b082597-6fd3-484a-9e6b-d9d400d0921a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.677826 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.677821    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvbw\" (UniqueName: \"kubernetes.io/projected/2b082597-6fd3-484a-9e6b-d9d400d0921a-kube-api-access-plvbw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.678147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.678124    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b082597-6fd3-484a-9e6b-d9d400d0921a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.698168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.698132    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvbw\" (UniqueName: \"kubernetes.io/projected/2b082597-6fd3-484a-9e6b-d9d400d0921a-kube-api-access-plvbw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-2xn6x\" (UID: \"2b082597-6fd3-484a-9e6b-d9d400d0921a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:29.735811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:29.735770    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:30.067801 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:30.067773    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"]
Oct 30 03:40:30.069471 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:40:30.069448    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b082597_6fd3_484a_9e6b_d9d400d0921a.slice/crio-4d9d51f05c7dfa6fe7c82123c688576e815e962a0e8d5093f11b338c3273d92b WatchSource:0}: Error finding container 4d9d51f05c7dfa6fe7c82123c688576e815e962a0e8d5093f11b338c3273d92b: Status 404 returned error can't find the container with id 4d9d51f05c7dfa6fe7c82123c688576e815e962a0e8d5093f11b338c3273d92b
Oct 30 03:40:30.116648 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:30.116613    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x" event={"ID":"2b082597-6fd3-484a-9e6b-d9d400d0921a","Type":"ContainerStarted","Data":"4d9d51f05c7dfa6fe7c82123c688576e815e962a0e8d5093f11b338c3273d92b"}
Oct 30 03:40:31.123569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:31.123530    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x" event={"ID":"2b082597-6fd3-484a-9e6b-d9d400d0921a","Type":"ContainerStarted","Data":"269cb0f7fdf27516dbfed0fdb4572f88a0571f2b59e669b6415db19fa7caf872"}
Oct 30 03:40:31.123978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:31.123614    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:31.193845 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:31.193792    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x" podStartSLOduration=2.193778928 podStartE2EDuration="2.193778928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:31.178805385 +0000 UTC m=+3570.252976175" watchObservedRunningTime="2026-10-30 03:40:31.193778928 +0000 UTC m=+3570.267949720"
Oct 30 03:40:39.162915 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:40:39.162878    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:40:42.131700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:42.131671    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-2xn6x"
Oct 30 03:40:46.025657 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:46.025624    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"]
Oct 30 03:40:46.038244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:46.038207    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kx9tq"]
Oct 30 03:40:47.065325 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:47.065292    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1c3afc-ce80-4e32-a686-ce6226f4d71a" path="/var/lib/kubelet/pods/cc1c3afc-ce80-4e32-a686-ce6226f4d71a/volumes"
Oct 30 03:40:53.450371 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.450330    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"]
Oct 30 03:40:53.454265 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.454222    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.473134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.473104    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"]
Oct 30 03:40:53.483168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.483137    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:40:53.483168 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.483156    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:40:53.535749 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.535715    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf76g\" (UniqueName: \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.535940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.535780    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.535940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.535878    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.636424 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.636389    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf76g\" (UniqueName: \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.636573 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.636442    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.636633 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.636562    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.637169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.637149    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.638829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.638807    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.656216 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.656187    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf76g\" (UniqueName: \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\") pod \"maas-api-revoke-sub-d576e68b-2h2dm\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:53.767377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:53.767297    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:54.067464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:54.067432    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"]
Oct 30 03:40:54.068984 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:40:54.068952    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95de62b6_e6ab_4a1b_a3a0_1e68fef81774.slice/crio-f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf WatchSource:0}: Error finding container f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf: Status 404 returned error can't find the container with id f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf
Oct 30 03:40:54.263434 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:54.263402    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm" event={"ID":"95de62b6-e6ab-4a1b-a3a0-1e68fef81774","Type":"ContainerStarted","Data":"f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf"}
Oct 30 03:40:55.270860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:55.270766    4214 generic.go:356] "Generic (PLEG): container finished" podID="95de62b6-e6ab-4a1b-a3a0-1e68fef81774" containerID="d34917a4c203fedb9c760176a3f077a6db7aa1477d05488e536e6bc8deb8265c" exitCode=0
Oct 30 03:40:55.271264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:55.270855    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm" event={"ID":"95de62b6-e6ab-4a1b-a3a0-1e68fef81774","Type":"ContainerDied","Data":"d34917a4c203fedb9c760176a3f077a6db7aa1477d05488e536e6bc8deb8265c"}
Oct 30 03:40:56.407624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.407596    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:56.463579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.463548    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\") pod \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") "
Oct 30 03:40:56.463762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.463601    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\" (UniqueName: \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\") pod \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") "
Oct 30 03:40:56.463762 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.463619    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\") pod \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\" (UID: \"95de62b6-e6ab-4a1b-a3a0-1e68fef81774\") "
Oct 30 03:40:56.464064 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.464039    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca" pod "95de62b6-e6ab-4a1b-a3a0-1e68fef81774" (UID: "95de62b6-e6ab-4a1b-a3a0-1e68fef81774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:56.465722 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.465694    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca" pod "95de62b6-e6ab-4a1b-a3a0-1e68fef81774" (UID: "95de62b6-e6ab-4a1b-a3a0-1e68fef81774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:56.465828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.465729    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g" pod "95de62b6-e6ab-4a1b-a3a0-1e68fef81774" (UID: "95de62b6-e6ab-4a1b-a3a0-1e68fef81774"). InnerVolumeSpecName "kube-api-access-zf76g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:56.564816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.564768    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf76g\" (UniqueName: \"kubernetes.io/projected/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-kube-api-access-zf76g\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:40:56.564816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.564813    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:40:56.564816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:56.564825    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95de62b6-e6ab-4a1b-a3a0-1e68fef81774-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:40:57.282837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:57.282800    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm" event={"ID":"95de62b6-e6ab-4a1b-a3a0-1e68fef81774","Type":"ContainerDied","Data":"f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf"}
Oct 30 03:40:57.283016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:57.282840    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"
Oct 30 03:40:57.283016 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:40:57.282857    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cc7ac51a107f58d8823cefa996060993e142e66e7cf907a7b3d15e748cbbbf"
Oct 30 03:41:07.822355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:07.822316    4214 scope.go:122] "RemoveContainer" containerID="68ef0b18ebabb1874ee4d87b4cf67941eb3300923325142aeb1e2807d0c830d2"
Oct 30 03:41:07.835128 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:07.835079    4214 scope.go:122] "RemoveContainer" containerID="7f5ddc74de174e726ad3614cf8594083cbb9a7b9aa7a237a3b6c887f02f60be4"
Oct 30 03:41:07.853346 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:07.853312    4214 scope.go:122] "RemoveContainer" containerID="c06f2c4b2db5059c5a209d0ae314309e4d85b2bdc384c159e9a47ae0a8445f35"
Oct 30 03:41:08.025916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:08.025879    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"]
Oct 30 03:41:08.040595 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:08.040552    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gh6ks"]
Oct 30 03:41:09.102303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:09.102262    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b3daa7-02d3-4652-8d2f-881d3b42f121" path="/var/lib/kubelet/pods/20b3daa7-02d3-4652-8d2f-881d3b42f121/volumes"
Oct 30 03:41:09.253783 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:41:09.253740    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:41:10.199448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.199411    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"]
Oct 30 03:41:10.199981 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.199956    4214 cpu_manager.go:414] "containerMap: removing container" podUID="95de62b6-e6ab-4a1b-a3a0-1e68fef81774" containerName="revoke-keys"
Oct 30 03:41:10.200035 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.199983    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95de62b6-e6ab-4a1b-a3a0-1e68fef81774" containerName="revoke-keys"
Oct 30 03:41:10.200094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.200084    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="95de62b6-e6ab-4a1b-a3a0-1e68fef81774" containerName="revoke-keys"
Oct 30 03:41:10.204956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.204937    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.221026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.220995    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"]
Oct 30 03:41:10.230511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.230488    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:41:10.230802 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.230780    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:41:10.234192 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.234170    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.234339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.234314    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9zq\" (UniqueName: \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.234473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.234455    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.335277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.335204    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.335464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.335298    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.335464 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.335346    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9zq\" (UniqueName: \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.335992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.335970    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.337513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.337491    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.355457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.355432    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9zq\" (UniqueName: \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\") pod \"maas-api-revoke-sub-d2bc93bc-4ghmz\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.518253 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.518152    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:10.824538 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:10.824506    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"]
Oct 30 03:41:10.825638 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:41:10.825611    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc094906a_e596_4365_b96d_c370af503668.slice/crio-0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3 WatchSource:0}: Error finding container 0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3: Status 404 returned error can't find the container with id 0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3
Oct 30 03:41:11.420706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:11.420613    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz" event={"ID":"c094906a-e596-4365-b96d-c370af503668","Type":"ContainerStarted","Data":"012b7fb80d904a1957f97953e5e8aef349a9dd41d5c989d7d4acf1c51cd55f95"}
Oct 30 03:41:11.420706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:11.420675    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz" event={"ID":"c094906a-e596-4365-b96d-c370af503668","Type":"ContainerStarted","Data":"0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3"}
Oct 30 03:41:11.461609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:11.461554    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz" podStartSLOduration=1.461537731 podStartE2EDuration="1.461537731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:11.460058821 +0000 UTC m=+3610.534229616" watchObservedRunningTime="2026-10-30 03:41:11.461537731 +0000 UTC m=+3610.535708526"
Oct 30 03:41:12.427719 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:12.427682    4214 generic.go:356] "Generic (PLEG): container finished" podID="c094906a-e596-4365-b96d-c370af503668" containerID="012b7fb80d904a1957f97953e5e8aef349a9dd41d5c989d7d4acf1c51cd55f95" exitCode=0
Oct 30 03:41:12.428256 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:12.427764    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz" event={"ID":"c094906a-e596-4365-b96d-c370af503668","Type":"ContainerDied","Data":"012b7fb80d904a1957f97953e5e8aef349a9dd41d5c989d7d4acf1c51cd55f95"}
Oct 30 03:41:13.567975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.567948    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:13.669778 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.669677    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\" (UniqueName: \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\") pod \"c094906a-e596-4365-b96d-c370af503668\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") "
Oct 30 03:41:13.669969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.669894    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\") pod \"c094906a-e596-4365-b96d-c370af503668\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") "
Oct 30 03:41:13.669969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.669946    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\") pod \"c094906a-e596-4365-b96d-c370af503668\" (UID: \"c094906a-e596-4365-b96d-c370af503668\") "
Oct 30 03:41:13.670370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.670345    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca" pod "c094906a-e596-4365-b96d-c370af503668" (UID: "c094906a-e596-4365-b96d-c370af503668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:13.671746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.671721    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq" pod "c094906a-e596-4365-b96d-c370af503668" (UID: "c094906a-e596-4365-b96d-c370af503668"). InnerVolumeSpecName "kube-api-access-mq9zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:13.671947 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.671930    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca" pod "c094906a-e596-4365-b96d-c370af503668" (UID: "c094906a-e596-4365-b96d-c370af503668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:13.771856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.771768    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094906a-e596-4365-b96d-c370af503668-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:41:13.771856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.771799    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094906a-e596-4365-b96d-c370af503668-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:41:13.771856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:13.771810    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9zq\" (UniqueName: \"kubernetes.io/projected/c094906a-e596-4365-b96d-c370af503668-kube-api-access-mq9zq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:41:14.440787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:14.440757    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"
Oct 30 03:41:14.440996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:14.440749    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz" event={"ID":"c094906a-e596-4365-b96d-c370af503668","Type":"ContainerDied","Data":"0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3"}
Oct 30 03:41:14.440996 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:14.440888    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0068c3978515e489e2edc6ca14e055a574b841496990d7ac91c930d4b71835f3"
Oct 30 03:41:22.407008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.406959    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:41:22.407559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.407539    4214 cpu_manager.go:414] "containerMap: removing container" podUID="c094906a-e596-4365-b96d-c370af503668" containerName="revoke-keys"
Oct 30 03:41:22.407559 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.407560    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094906a-e596-4365-b96d-c370af503668" containerName="revoke-keys"
Oct 30 03:41:22.407670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.407660    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="c094906a-e596-4365-b96d-c370af503668" containerName="revoke-keys"
Oct 30 03:41:22.411189 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.411169    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.427515 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.427479    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-70abcb-openshift-default-dockercfg-vjspj\""
Oct 30 03:41:22.443341 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.443304    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:41:22.563116 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563069    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563135    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563199    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563322    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563388    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7hp\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563426    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563455    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563485    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.563554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.563550    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664531    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664579    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664618 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664605    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7hp\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664634    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664663    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664682    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664718    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664758    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.664945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.664825    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.665330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.665002    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.665330 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.665070    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.665442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.665361    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.665442 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.665426    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.665517 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.665477    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.666924 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.666905    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.667187 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.667168    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.684874 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.684844    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.685058 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.685038    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7hp\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\") pod \"e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") " pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:22.733285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:22.733254    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:23.055351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.055324    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:41:23.056187 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:41:23.056161    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db085ad_d231_448a_aa57_093180e0c737.slice/crio-01054a2a0213644938cdee553bb21a9f59074c2e01d7c71c3810a335b7171c86 WatchSource:0}: Error finding container 01054a2a0213644938cdee553bb21a9f59074c2e01d7c71c3810a335b7171c86: Status 404 returned error can't find the container with id 01054a2a0213644938cdee553bb21a9f59074c2e01d7c71c3810a335b7171c86
Oct 30 03:41:23.058564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.058527    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:23.058651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.058611    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:23.058691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.058653    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:23.495754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.495677    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerStarted","Data":"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"}
Oct 30 03:41:23.495754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.495728    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerStarted","Data":"01054a2a0213644938cdee553bb21a9f59074c2e01d7c71c3810a335b7171c86"}
Oct 30 03:41:23.551706 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.551649    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podStartSLOduration=1.551636292 podStartE2EDuration="1.551636292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:23.550244775 +0000 UTC m=+3622.624415559" watchObservedRunningTime="2026-10-30 03:41:23.551636292 +0000 UTC m=+3622.625807084"
Oct 30 03:41:23.734001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:23.733972    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:24.738788 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:24.738761    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:25.511175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:25.511140    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:25.512021 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:25.511999    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:27.976690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:27.976657    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:41:27.980467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:27.980444    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.037668 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.037633    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-70abcb\""
Oct 30 03:41:28.037904 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.037874    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:41:28.094351 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.094318    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b05be57c-dockercfg-xlrd9\""
Oct 30 03:41:28.126335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.126303    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9g58\" (UniqueName: \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.126511 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.126366    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.227670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.227590    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9g58\" (UniqueName: \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.227670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.227651    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.228399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.228374    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.248471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.248440    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9g58\" (UniqueName: \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\") pod \"payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.348886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.348856    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:28.647169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:28.647145    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:41:28.648897 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:41:28.648856    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccf14d9_b73f_4202_a4d1_91a648d725cd.slice/crio-0bf8ce0df6fab87db77e2973cb7f9dce074b3c3902f68845e222e29a43f32ccb WatchSource:0}: Error finding container 0bf8ce0df6fab87db77e2973cb7f9dce074b3c3902f68845e222e29a43f32ccb: Status 404 returned error can't find the container with id 0bf8ce0df6fab87db77e2973cb7f9dce074b3c3902f68845e222e29a43f32ccb
Oct 30 03:41:29.027811 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.027731    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"]
Oct 30 03:41:29.040334 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.040302    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzmh8"]
Oct 30 03:41:29.065322 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.065289    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8d0d45-ae73-410a-8a6e-f7db870ffa0c" path="/var/lib/kubelet/pods/2c8d0d45-ae73-410a-8a6e-f7db870ffa0c/volumes"
Oct 30 03:41:29.536051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.536011    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" event={"ID":"bccf14d9-b73f-4202-a4d1-91a648d725cd","Type":"ContainerStarted","Data":"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"}
Oct 30 03:41:29.536272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.536070    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" event={"ID":"bccf14d9-b73f-4202-a4d1-91a648d725cd","Type":"ContainerStarted","Data":"0bf8ce0df6fab87db77e2973cb7f9dce074b3c3902f68845e222e29a43f32ccb"}
Oct 30 03:41:29.536272 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.536215    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:29.581886 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:29.581792    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" podStartSLOduration=2.58176811 podStartE2EDuration="2.58176811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:29.576911464 +0000 UTC m=+3628.651082264" watchObservedRunningTime="2026-10-30 03:41:29.58176811 +0000 UTC m=+3628.655938904"
Oct 30 03:41:39.164601 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:41:39.164560    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:41:40.545912 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:40.545881    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:41:40.956709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:40.956371    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": read tcp 10.133.0.2:43068->10.133.1.69:15021: read: connection reset by peer" start-of-body=
Oct 30 03:41:40.956709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:40.956451    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": read tcp 10.133.0.2:43068->10.133.1.69:15021: read: connection reset by peer"
Oct 30 03:41:41.629992 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.629959    4214 generic.go:356] "Generic (PLEG): container finished" podID="5db085ad-d231-448a-aa57-093180e0c737" containerID="41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38" exitCode=0
Oct 30 03:41:41.630463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.630034    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerDied","Data":"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"}
Oct 30 03:41:41.630463 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.630447    4214 scope.go:122] "RemoveContainer" containerID="41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"
Oct 30 03:41:41.631670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.631640    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:41.631782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.631728    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:41.631782 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:41.631775    4214 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:42.655254 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:42.643073    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerStarted","Data":"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"}
Oct 30 03:41:42.758251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:42.757660    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:42.758251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:42.757789    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 03:41:42.758251 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:42.757846    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 03:41:43.738918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:43.738327    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 03:41:43.738918 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:43.738387    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 03:41:44.733846 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:44.733811    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 03:41:44.734069 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:44.733888    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 03:41:45.734178 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:45.734146    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 03:41:45.734745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:45.734196    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 03:41:46.734395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:46.734361    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 03:41:46.734994 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:46.734963    4214 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 03:41:48.206655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:48.206619    4214 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:48.207303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:48.207278    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:48.234856 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:48.234827    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:41:50.026745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:50.026714    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"]
Oct 30 03:41:50.039745 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:50.039720    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s6z4x"]
Oct 30 03:41:51.065790 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:41:51.065744    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db11e4e2-9d17-475c-bd94-2e044c6539e5" path="/var/lib/kubelet/pods/db11e4e2-9d17-475c-bd94-2e044c6539e5/volumes"
Oct 30 03:42:02.766487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.766442    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"]
Oct 30 03:42:02.770519 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.770494    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.789908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.789880    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"]
Oct 30 03:42:02.798954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.798925    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:42:02.798954 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.798945    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:42:02.826134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.826105    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.826303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.826190    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lcbs\" (UniqueName: \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.826303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.826282    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.926872 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.926839    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.927047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.926935    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.927047 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.926972    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lcbs\" (UniqueName: \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.927587 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.927565    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.929285 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.929261    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:02.947979 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:02.947948    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lcbs\" (UniqueName: \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\") pod \"maas-api-revoke-sub-efda0460-klbfv\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:03.083134 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:03.083106    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:03.390972 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:03.390947    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"]
Oct 30 03:42:03.392939 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:42:03.392909    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e70755_392b_4b77_8249_b0d7aa5d67b3.slice/crio-e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d WatchSource:0}: Error finding container e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d: Status 404 returned error can't find the container with id e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d
Oct 30 03:42:03.802257 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:03.802191    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv" event={"ID":"83e70755-392b-4b77-8249-b0d7aa5d67b3","Type":"ContainerStarted","Data":"49628de4ca87d7f2cfbad8b3ab35c29f756175b6dc309417aa652f89c463d42d"}
Oct 30 03:42:03.802675 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:03.802308    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv" event={"ID":"83e70755-392b-4b77-8249-b0d7aa5d67b3","Type":"ContainerStarted","Data":"e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d"}
Oct 30 03:42:03.844335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:03.844282    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv" podStartSLOduration=1.844268151 podStartE2EDuration="1.844268151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:03.842530012 +0000 UTC m=+3662.916700816" watchObservedRunningTime="2026-10-30 03:42:03.844268151 +0000 UTC m=+3662.918438944"
Oct 30 03:42:04.809850 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:04.809814    4214 generic.go:356] "Generic (PLEG): container finished" podID="83e70755-392b-4b77-8249-b0d7aa5d67b3" containerID="49628de4ca87d7f2cfbad8b3ab35c29f756175b6dc309417aa652f89c463d42d" exitCode=0
Oct 30 03:42:04.810273 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:04.809905    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv" event={"ID":"83e70755-392b-4b77-8249-b0d7aa5d67b3","Type":"ContainerDied","Data":"49628de4ca87d7f2cfbad8b3ab35c29f756175b6dc309417aa652f89c463d42d"}
Oct 30 03:42:05.949046 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.949021    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:05.975758 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.975720    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\") pod \"83e70755-392b-4b77-8249-b0d7aa5d67b3\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") "
Oct 30 03:42:05.975919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.975861    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\" (UniqueName: \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\") pod \"83e70755-392b-4b77-8249-b0d7aa5d67b3\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") "
Oct 30 03:42:05.975919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.975885    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\") pod \"83e70755-392b-4b77-8249-b0d7aa5d67b3\" (UID: \"83e70755-392b-4b77-8249-b0d7aa5d67b3\") "
Oct 30 03:42:05.976357 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.976326    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca" pod "83e70755-392b-4b77-8249-b0d7aa5d67b3" (UID: "83e70755-392b-4b77-8249-b0d7aa5d67b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:05.977921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.977894    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca" pod "83e70755-392b-4b77-8249-b0d7aa5d67b3" (UID: "83e70755-392b-4b77-8249-b0d7aa5d67b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:05.978043 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:05.977997    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs" pod "83e70755-392b-4b77-8249-b0d7aa5d67b3" (UID: "83e70755-392b-4b77-8249-b0d7aa5d67b3"). InnerVolumeSpecName "kube-api-access-2lcbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:06.076869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.076778    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lcbs\" (UniqueName: \"kubernetes.io/projected/83e70755-392b-4b77-8249-b0d7aa5d67b3-kube-api-access-2lcbs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:06.076869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.076805    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:06.076869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.076818    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e70755-392b-4b77-8249-b0d7aa5d67b3-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:06.824729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.824699    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"
Oct 30 03:42:06.824921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.824729    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv" event={"ID":"83e70755-392b-4b77-8249-b0d7aa5d67b3","Type":"ContainerDied","Data":"e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d"}
Oct 30 03:42:06.824921 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:06.824776    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28ae0c8fc9836311c19f7521c1e17cb67ceef00faed2b17e4d2e17b66997d9d"
Oct 30 03:42:07.892619 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:07.892584    4214 scope.go:122] "RemoveContainer" containerID="95311ef351d8bbe1df49c30d01a19a7193cd922d0cc37a77ec904f586726debe"
Oct 30 03:42:07.901911 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:07.901893    4214 scope.go:122] "RemoveContainer" containerID="5b30f30f324ecf303f26a2fbc6b6434df1684c972e6233902b80cdc404cf9502"
Oct 30 03:42:07.910690 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:07.910670    4214 scope.go:122] "RemoveContainer" containerID="cb35e727b9c93ffd8ff2ef0fd076952d5e4c74d7876f220482c8332acdd42802"
Oct 30 03:42:07.918480 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:07.918463    4214 scope.go:122] "RemoveContainer" containerID="033f84b0393eaa5f2449c1d43561bba5a39cf1fc8c7f4540c689add6d5dd0c92"
Oct 30 03:42:09.163388 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:42:09.163347    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:42:13.696751 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.696709    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"]
Oct 30 03:42:13.697183 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.697169    4214 cpu_manager.go:414] "containerMap: removing container" podUID="83e70755-392b-4b77-8249-b0d7aa5d67b3" containerName="revoke-keys"
Oct 30 03:42:13.697290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.697184    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e70755-392b-4b77-8249-b0d7aa5d67b3" containerName="revoke-keys"
Oct 30 03:42:13.697290 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.697267    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e70755-392b-4b77-8249-b0d7aa5d67b3" containerName="revoke-keys"
Oct 30 03:42:13.702169 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.702153    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.725184 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.725144    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"]
Oct 30 03:42:13.735439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.735406    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:42:13.736391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.736368    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:42:13.758723 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.758694    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lpp\" (UniqueName: \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.759080 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.759053    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.759223 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.759205    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.860915 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.860871    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.861118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.860929    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.861118 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.861051    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lpp\" (UniqueName: \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.861612 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.861579    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.863647 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.863619    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:13.882796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:13.882768    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lpp\" (UniqueName: \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:14.029653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:14.029565    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:14.350200 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:14.350168    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"]
Oct 30 03:42:14.352722 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:42:14.352687    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74171ae3_0abf_4ae7_8cc9_3c402f4a8cc1.slice/crio-02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65 WatchSource:0}: Error finding container 02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65: Status 404 returned error can't find the container with id 02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65
Oct 30 03:42:14.881969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:14.881930    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f" event={"ID":"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1","Type":"ContainerStarted","Data":"914cdd9e176caf1ec1e58559a47de22c7fd9dc5f8948550859e2fa87b72fad13"}
Oct 30 03:42:14.882396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:14.881995    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f" event={"ID":"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1","Type":"ContainerStarted","Data":"02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65"}
Oct 30 03:42:14.923075 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:14.922974    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f" podStartSLOduration=1.922959095 podStartE2EDuration="1.922959095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:14.920331609 +0000 UTC m=+3673.994502440" watchObservedRunningTime="2026-10-30 03:42:14.922959095 +0000 UTC m=+3673.997129888"
Oct 30 03:42:15.889487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:15.889449    4214 generic.go:356] "Generic (PLEG): container finished" podID="74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" containerID="914cdd9e176caf1ec1e58559a47de22c7fd9dc5f8948550859e2fa87b72fad13" exitCode=0
Oct 30 03:42:15.890051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:15.889543    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f" event={"ID":"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1","Type":"ContainerDied","Data":"914cdd9e176caf1ec1e58559a47de22c7fd9dc5f8948550859e2fa87b72fad13"}
Oct 30 03:42:17.031087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.031063    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:17.093764 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.093726    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\") pod \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") "
Oct 30 03:42:17.093931 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.093909    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\") pod \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") "
Oct 30 03:42:17.093977 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.093948    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\" (UniqueName: \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\") pod \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\" (UID: \"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1\") "
Oct 30 03:42:17.094297 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.094275    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca" pod "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" (UID: "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:17.095860 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.095832    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca" pod "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" (UID: "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:17.096015 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.095992    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp" pod "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" (UID: "74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1"). InnerVolumeSpecName "kube-api-access-45lpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:17.195760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.195668    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:17.195760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.195701    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:17.195760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.195718    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lpp\" (UniqueName: \"kubernetes.io/projected/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1-kube-api-access-45lpp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:17.903051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.903020    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f" event={"ID":"74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1","Type":"ContainerDied","Data":"02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65"}
Oct 30 03:42:17.903051 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.903046    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"
Oct 30 03:42:17.903307 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:17.903075    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02314ca2d1478f89920e7c0c400aebb0107b4b73888602a6dc26a815001eeb65"
Oct 30 03:42:24.111971 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:24.111928    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:42:25.620711 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:25.620676    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"]
Oct 30 03:42:25.637247 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:25.637194    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-70abcb-95h7f"]
Oct 30 03:42:27.066103 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:27.066066    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" path="/var/lib/kubelet/pods/74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1/volumes"
Oct 30 03:42:29.112692 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:29.112652    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing" containerID="cri-o://8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef" gracePeriod=25
Oct 30 03:42:30.544554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:30.544516    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:42:32.117042 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:32.117004    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:42:32.117620 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:32.117293    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" containerID="cri-o://8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1" gracePeriod=30
Oct 30 03:42:33.209362 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:33.209329    4214 patch_prober.go:29] interesting pod/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:42:33.209725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:33.209395    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:42:37.366408 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.366382    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:42:37.411523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411463    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411523 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411519    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411545    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411587    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411611    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411630    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411688    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.411728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411715    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.412087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411746    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\") pod \"5db085ad-d231-448a-aa57-093180e0c737\" (UID: \"5db085ad-d231-448a-aa57-093180e0c737\") "
Oct 30 03:42:37.412087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.411761    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:37.412087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.412014    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:37.412087 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.412022    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:37.412335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.412088    4214 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.412335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.412152    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:37.412335 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.412277    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:37.413816 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.413793    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "kube-api-access-jw7hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:37.413945 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.413927    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:37.414177 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.414157    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:37.414303 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.414180    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo" pod "5db085ad-d231-448a-aa57-093180e0c737" (UID: "5db085ad-d231-448a-aa57-093180e0c737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:42:37.513395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513362    4214 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-data\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513390    4214 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-istio-envoy\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513398    4214 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-istio-token\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513408    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7hp\" (UniqueName: \"kubernetes.io/projected/5db085ad-d231-448a-aa57-093180e0c737-kube-api-access-jw7hp\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513421    4214 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-workload-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513434    4214 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db085ad-d231-448a-aa57-093180e0c737-istiod-ca-cert\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513446    4214 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db085ad-d231-448a-aa57-093180e0c737-credential-socket\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:37.513638 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:37.513458    4214 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db085ad-d231-448a-aa57-093180e0c737-istio-podinfo\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:38.029944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.029905    4214 generic.go:356] "Generic (PLEG): container finished" podID="5db085ad-d231-448a-aa57-093180e0c737" containerID="8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1" exitCode=0
Oct 30 03:42:38.030126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.029948    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerDied","Data":"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"}
Oct 30 03:42:38.030126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.029957    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"
Oct 30 03:42:38.030126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.030004    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74" event={"ID":"5db085ad-d231-448a-aa57-093180e0c737","Type":"ContainerDied","Data":"01054a2a0213644938cdee553bb21a9f59074c2e01d7c71c3810a335b7171c86"}
Oct 30 03:42:38.030126 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.030033    4214 scope.go:122] "RemoveContainer" containerID="8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"
Oct 30 03:42:38.040347 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.040328    4214 scope.go:122] "RemoveContainer" containerID="41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"
Oct 30 03:42:38.049174 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.049154    4214 scope.go:122] "RemoveContainer" containerID="8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"
Oct 30 03:42:38.049471 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:42:38.049452    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1\": container with ID starting with 8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1 not found: ID does not exist" containerID="8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"
Oct 30 03:42:38.049556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.049484    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1"} err="failed to get container status \"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1\": rpc error: code = NotFound desc = could not find container \"8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1\": container with ID starting with 8a8e9100548626c65cd131d0592397883c9b30a5b0a13878ba3a6e17afd959d1 not found: ID does not exist"
Oct 30 03:42:38.049556 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.049508    4214 scope.go:122] "RemoveContainer" containerID="41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"
Oct 30 03:42:38.049752 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:42:38.049731    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38\": container with ID starting with 41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38 not found: ID does not exist" containerID="41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"
Oct 30 03:42:38.049796 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.049760    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38"} err="failed to get container status \"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38\": rpc error: code = NotFound desc = could not find container \"41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38\": container with ID starting with 41f5d6a4972452fc744996a4e8e868dad4af1963371541c4e5babe7280630f38 not found: ID does not exist"
Oct 30 03:42:38.097210 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.097175    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:42:38.101441 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:38.101418    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-70abcb-openshift-default-5d58c5bc49qtt74"]
Oct 30 03:42:39.065472 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:39.065436    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db085ad-d231-448a-aa57-093180e0c737" path="/var/lib/kubelet/pods/5db085ad-d231-448a-aa57-093180e0c737/volumes"
Oct 30 03:42:39.164484 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:42:39.164449    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:42:40.544703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:40.544662    4214 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:42:49.364736 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.364709    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:42:49.477342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.477219    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\") pod \"bccf14d9-b73f-4202-a4d1-91a648d725cd\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") "
Oct 30 03:42:49.477508 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.477404    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\" (UniqueName: \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\") pod \"bccf14d9-b73f-4202-a4d1-91a648d725cd\" (UID: \"bccf14d9-b73f-4202-a4d1-91a648d725cd\") "
Oct 30 03:42:49.477683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.477661    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume" pod "bccf14d9-b73f-4202-a4d1-91a648d725cd" (UID: "bccf14d9-b73f-4202-a4d1-91a648d725cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:49.477775 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.477762    4214 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccf14d9-b73f-4202-a4d1-91a648d725cd-plugins-config-volume\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:49.479503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.479481    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58" pod "bccf14d9-b73f-4202-a4d1-91a648d725cd" (UID: "bccf14d9-b73f-4202-a4d1-91a648d725cd"). InnerVolumeSpecName "kube-api-access-p9g58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:49.578717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:49.578686    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9g58\" (UniqueName: \"kubernetes.io/projected/bccf14d9-b73f-4202-a4d1-91a648d725cd-kube-api-access-p9g58\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:42:50.144505 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.144459    4214 generic.go:356] "Generic (PLEG): container finished" podID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerID="8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef" exitCode=0
Oct 30 03:42:50.144718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.144513    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"
Oct 30 03:42:50.144718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.144548    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" event={"ID":"bccf14d9-b73f-4202-a4d1-91a648d725cd","Type":"ContainerDied","Data":"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"}
Oct 30 03:42:50.144718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.144632    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c" event={"ID":"bccf14d9-b73f-4202-a4d1-91a648d725cd","Type":"ContainerDied","Data":"0bf8ce0df6fab87db77e2973cb7f9dce074b3c3902f68845e222e29a43f32ccb"}
Oct 30 03:42:50.144718 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.144668    4214 scope.go:122] "RemoveContainer" containerID="8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"
Oct 30 03:42:50.154727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.154707    4214 scope.go:122] "RemoveContainer" containerID="8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"
Oct 30 03:42:50.154999 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:42:50.154961    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef\": container with ID starting with 8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef not found: ID does not exist" containerID="8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"
Oct 30 03:42:50.155071 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.155007    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef"} err="failed to get container status \"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef\": rpc error: code = NotFound desc = could not find container \"8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef\": container with ID starting with 8a11b847cce5c159852b0006c20c7a51af3247564791ffed710142f83f7021ef not found: ID does not exist"
Oct 30 03:42:50.197579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.197547    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:42:50.210450 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:50.210424    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-70abcb-64bd795f7xs97c"]
Oct 30 03:42:51.066942 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:42:51.066909    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" path="/var/lib/kubelet/pods/bccf14d9-b73f-4202-a4d1-91a648d725cd/volumes"
Oct 30 03:43:07.953913 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:07.953882    4214 scope.go:122] "RemoveContainer" containerID="6cca668307d7ba32507c0d2064e8aa36c17f9f4a30e3874e4494284b3d4e3907"
Oct 30 03:43:07.962787 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:07.962764    4214 scope.go:122] "RemoveContainer" containerID="f26a39a67563d953fcff96747288643508da0cd4d7ebf520a99896aceb864370"
Oct 30 03:43:07.971468 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:07.971443    4214 scope.go:122] "RemoveContainer" containerID="56fdf20c3e26feb17cc4b2a41b6da011be1e16cfca51b5c2a45cbaec944e6d9a"
Oct 30 03:43:09.167715 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:43:09.167661    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:43:12.466694 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.466657    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"]
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467086    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467099    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467113    4214 cpu_manager.go:414] "containerMap: removing container" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467119    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467125    4214 cpu_manager.go:414] "containerMap: removing container" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467130    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467143    4214 cpu_manager.go:414] "containerMap: removing container" podUID="74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" containerName="revoke-keys"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467148    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" containerName="revoke-keys"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467219    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467241    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db085ad-d231-448a-aa57-093180e0c737" containerName="istio-proxy"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467251    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccf14d9-b73f-4202-a4d1-91a648d725cd" containerName="payload-processing"
Oct 30 03:43:12.467263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.467261    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="74171ae3-0abf-4ae7-8cc9-3c402f4a8cc1" containerName="revoke-keys"
Oct 30 03:43:12.472191 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.472170    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.488413 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.488383    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"]
Oct 30 03:43:12.498956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.498927    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:43:12.499127 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.499028    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:43:12.530520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.530184    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hkj\" (UniqueName: \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.530520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.530312    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.530520 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.530367    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.631746 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.631700    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.631965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.631802    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.631965 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.631924    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hkj\" (UniqueName: \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.632547 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.632508    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.634530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.634501    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.664879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.664841    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hkj\" (UniqueName: \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\") pod \"maas-api-revoke-sub-452b6d7b-p7x8q\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:12.811300 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:12.803408    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:13.423219 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:13.423177    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"]
Oct 30 03:43:14.409743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:14.409381    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q" event={"ID":"7a3774b3-c144-4016-8262-b887353bcdbb","Type":"ContainerStarted","Data":"98e71376e0f68599f43c2dae7057dc7f2758064f1d6f1bc774118ed84a2e2701"}
Oct 30 03:43:14.409743 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:14.409485    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q" event={"ID":"7a3774b3-c144-4016-8262-b887353bcdbb","Type":"ContainerStarted","Data":"7988f0825ceab6252e804568de66423092850ba397ba1e335ce4828950935253"}
Oct 30 03:43:14.475466 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:14.461194    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q" podStartSLOduration=2.461172693 podStartE2EDuration="2.461172693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:14.450522337 +0000 UTC m=+3733.524693132" watchObservedRunningTime="2026-10-30 03:43:14.461172693 +0000 UTC m=+3733.535343486"
Oct 30 03:43:15.417627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:15.417592    4214 generic.go:356] "Generic (PLEG): container finished" podID="7a3774b3-c144-4016-8262-b887353bcdbb" containerID="98e71376e0f68599f43c2dae7057dc7f2758064f1d6f1bc774118ed84a2e2701" exitCode=0
Oct 30 03:43:15.418057 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:15.417687    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q" event={"ID":"7a3774b3-c144-4016-8262-b887353bcdbb","Type":"ContainerDied","Data":"98e71376e0f68599f43c2dae7057dc7f2758064f1d6f1bc774118ed84a2e2701"}
Oct 30 03:43:16.554142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.554109    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:16.585624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.585595    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\") pod \"7a3774b3-c144-4016-8262-b887353bcdbb\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") "
Oct 30 03:43:16.585771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.585667    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\") pod \"7a3774b3-c144-4016-8262-b887353bcdbb\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") "
Oct 30 03:43:16.585771 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.585723    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\" (UniqueName: \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\") pod \"7a3774b3-c144-4016-8262-b887353bcdbb\" (UID: \"7a3774b3-c144-4016-8262-b887353bcdbb\") "
Oct 30 03:43:16.585969 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.585946    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca" pod "7a3774b3-c144-4016-8262-b887353bcdbb" (UID: "7a3774b3-c144-4016-8262-b887353bcdbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:16.587737 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.587709    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca" pod "7a3774b3-c144-4016-8262-b887353bcdbb" (UID: "7a3774b3-c144-4016-8262-b887353bcdbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:16.587831 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.587778    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj" pod "7a3774b3-c144-4016-8262-b887353bcdbb" (UID: "7a3774b3-c144-4016-8262-b887353bcdbb"). InnerVolumeSpecName "kube-api-access-95hkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:16.687039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.686954    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:16.687039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.686991    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3774b3-c144-4016-8262-b887353bcdbb-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:16.687039 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:16.687001    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hkj\" (UniqueName: \"kubernetes.io/projected/7a3774b3-c144-4016-8262-b887353bcdbb-kube-api-access-95hkj\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:17.429640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:17.429558    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q"
Oct 30 03:43:17.429640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:17.429573    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p7x8q" event={"ID":"7a3774b3-c144-4016-8262-b887353bcdbb","Type":"ContainerDied","Data":"7988f0825ceab6252e804568de66423092850ba397ba1e335ce4828950935253"}
Oct 30 03:43:17.429640 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:17.429619    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7988f0825ceab6252e804568de66423092850ba397ba1e335ce4828950935253"
Oct 30 03:43:29.858882 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.858841    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"]
Oct 30 03:43:29.859569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.859472    4214 cpu_manager.go:414] "containerMap: removing container" podUID="7a3774b3-c144-4016-8262-b887353bcdbb" containerName="revoke-keys"
Oct 30 03:43:29.859569 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.859489    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3774b3-c144-4016-8262-b887353bcdbb" containerName="revoke-keys"
Oct 30 03:43:29.859703 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.859626    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3774b3-c144-4016-8262-b887353bcdbb" containerName="revoke-keys"
Oct 30 03:43:29.863525 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.863500    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:29.888367 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.888033    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"]
Oct 30 03:43:29.894497 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.894460    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:43:29.894754 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.894726    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:43:29.940414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.940295    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclkq\" (UniqueName: \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:29.940414 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.940361    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:29.940660 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:29.940525    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.041094 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.041059    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclkq\" (UniqueName: \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.041317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.041126    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.041317 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.041179    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.041928 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.041906    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.044209 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.044186    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.062399 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.062370    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclkq\" (UniqueName: \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\") pod \"maas-api-revoke-sub-3e31a506-pp4kk\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.185503 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.185407    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:30.518683 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.518598    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"]
Oct 30 03:43:30.526384 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:43:30.526350    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026fc7fa_dff5_4d51_b183_f8e313889c11.slice/crio-a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5 WatchSource:0}: Error finding container a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5: Status 404 returned error can't find the container with id a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5
Oct 30 03:43:30.541448 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:30.541416    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk" event={"ID":"026fc7fa-dff5-4d51-b183-f8e313889c11","Type":"ContainerStarted","Data":"a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5"}
Oct 30 03:43:31.575147 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:31.575087    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk" event={"ID":"026fc7fa-dff5-4d51-b183-f8e313889c11","Type":"ContainerStarted","Data":"5fec8ac033fada0fb91bc717814ca417aea10749855c13814e23377bee03cb97"}
Oct 30 03:43:31.625211 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:31.625152    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk" podStartSLOduration=2.625139336 podStartE2EDuration="2.625139336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:31.622166194 +0000 UTC m=+3750.696336987" watchObservedRunningTime="2026-10-30 03:43:31.625139336 +0000 UTC m=+3750.699310129"
Oct 30 03:43:32.582240 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:32.582191    4214 generic.go:356] "Generic (PLEG): container finished" podID="026fc7fa-dff5-4d51-b183-f8e313889c11" containerID="5fec8ac033fada0fb91bc717814ca417aea10749855c13814e23377bee03cb97" exitCode=0
Oct 30 03:43:32.582709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:32.582277    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk" event={"ID":"026fc7fa-dff5-4d51-b183-f8e313889c11","Type":"ContainerDied","Data":"5fec8ac033fada0fb91bc717814ca417aea10749855c13814e23377bee03cb97"}
Oct 30 03:43:33.726609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.726587    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:33.851395 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.851306    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\" (UniqueName: \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\") pod \"026fc7fa-dff5-4d51-b183-f8e313889c11\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") "
Oct 30 03:43:33.851554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.851426    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\") pod \"026fc7fa-dff5-4d51-b183-f8e313889c11\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") "
Oct 30 03:43:33.851554 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.851524    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\") pod \"026fc7fa-dff5-4d51-b183-f8e313889c11\" (UID: \"026fc7fa-dff5-4d51-b183-f8e313889c11\") "
Oct 30 03:43:33.851893 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.851861    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca" pod "026fc7fa-dff5-4d51-b183-f8e313889c11" (UID: "026fc7fa-dff5-4d51-b183-f8e313889c11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:33.853558 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.853530    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca" pod "026fc7fa-dff5-4d51-b183-f8e313889c11" (UID: "026fc7fa-dff5-4d51-b183-f8e313889c11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:33.853653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.853587    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq" pod "026fc7fa-dff5-4d51-b183-f8e313889c11" (UID: "026fc7fa-dff5-4d51-b183-f8e313889c11"). InnerVolumeSpecName "kube-api-access-hclkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:33.953020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.952984    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclkq\" (UniqueName: \"kubernetes.io/projected/026fc7fa-dff5-4d51-b183-f8e313889c11-kube-api-access-hclkq\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:33.953020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.953015    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:33.953020 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:33.953025    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026fc7fa-dff5-4d51-b183-f8e313889c11-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:34.595280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:34.595244    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk" event={"ID":"026fc7fa-dff5-4d51-b183-f8e313889c11","Type":"ContainerDied","Data":"a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5"}
Oct 30 03:43:34.595467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:34.595298    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4902387463dd08635b8a9cb42f1f2528487dcced1fd911651f8b2db494d75e5"
Oct 30 03:43:34.595467 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:34.595259    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-pp4kk"
Oct 30 03:43:39.163145 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:43:39.163101    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:43:40.029370 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:40.029338    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"]
Oct 30 03:43:40.041264 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:40.041204    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7957080-5fjhv"]
Oct 30 03:43:41.074481 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:41.074446    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7679e79-59c8-4169-a056-d48b087ab1b5" path="/var/lib/kubelet/pods/f7679e79-59c8-4169-a056-d48b087ab1b5/volumes"
Oct 30 03:43:52.444207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.444170    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"]
Oct 30 03:43:52.444624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.444609    4214 cpu_manager.go:414] "containerMap: removing container" podUID="026fc7fa-dff5-4d51-b183-f8e313889c11" containerName="revoke-keys"
Oct 30 03:43:52.444624 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.444622    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026fc7fa-dff5-4d51-b183-f8e313889c11" containerName="revoke-keys"
Oct 30 03:43:52.444725 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.444714    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="026fc7fa-dff5-4d51-b183-f8e313889c11" containerName="revoke-keys"
Oct 30 03:43:52.448471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.448455    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.470185 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.470153    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"]
Oct 30 03:43:52.479440 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.479409    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:43:52.479627 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.479448    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:43:52.542309 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.542271    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.542475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.542321    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.542475 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.542401    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77r4l\" (UniqueName: \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.643439 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.643396    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.643643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.643456    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.643643 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.643493    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77r4l\" (UniqueName: \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.644287 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.644257    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.645933 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.645904    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.664869 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.664839    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77r4l\" (UniqueName: \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\") pod \"maas-api-revoke-sub-f58a9c96-g5lks\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:52.761645 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:52.761567    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:53.065280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:53.065252    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"]
Oct 30 03:43:53.065775 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:43:53.065741    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdea09b_3e82_42c2_b7e9_02d6236111a2.slice/crio-58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1 WatchSource:0}: Error finding container 58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1: Status 404 returned error can't find the container with id 58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1
Oct 30 03:43:53.750879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:53.750782    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks" event={"ID":"3fdea09b-3e82-42c2-b7e9-02d6236111a2","Type":"ContainerStarted","Data":"3d272d2dd322ee7138fcbb9b97b68e48b38e338cabc524691fdf355f50ad18d2"}
Oct 30 03:43:53.750879 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:53.750838    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks" event={"ID":"3fdea09b-3e82-42c2-b7e9-02d6236111a2","Type":"ContainerStarted","Data":"58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1"}
Oct 30 03:43:53.793794 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:53.793734    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks" podStartSLOduration=1.793719631 podStartE2EDuration="1.793719631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:53.791324286 +0000 UTC m=+3772.865495080" watchObservedRunningTime="2026-10-30 03:43:53.793719631 +0000 UTC m=+3772.867890423"
Oct 30 03:43:54.757299 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:54.757265    4214 generic.go:356] "Generic (PLEG): container finished" podID="3fdea09b-3e82-42c2-b7e9-02d6236111a2" containerID="3d272d2dd322ee7138fcbb9b97b68e48b38e338cabc524691fdf355f50ad18d2" exitCode=0
Oct 30 03:43:54.757784 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:54.757308    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks" event={"ID":"3fdea09b-3e82-42c2-b7e9-02d6236111a2","Type":"ContainerDied","Data":"3d272d2dd322ee7138fcbb9b97b68e48b38e338cabc524691fdf355f50ad18d2"}
Oct 30 03:43:55.895455 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.895431    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:55.977670 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.977625    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\") pod \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") "
Oct 30 03:43:55.977837 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.977696    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\" (UniqueName: \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\") pod \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") "
Oct 30 03:43:55.977878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.977856    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\") pod \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\" (UID: \"3fdea09b-3e82-42c2-b7e9-02d6236111a2\") "
Oct 30 03:43:55.978024 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.978003    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca" pod "3fdea09b-3e82-42c2-b7e9-02d6236111a2" (UID: "3fdea09b-3e82-42c2-b7e9-02d6236111a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:55.978173 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.978159    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-openshift-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:55.980009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.979980    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca" pod "3fdea09b-3e82-42c2-b7e9-02d6236111a2" (UID: "3fdea09b-3e82-42c2-b7e9-02d6236111a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:55.980148 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:55.980041    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l" pod "3fdea09b-3e82-42c2-b7e9-02d6236111a2" (UID: "3fdea09b-3e82-42c2-b7e9-02d6236111a2"). InnerVolumeSpecName "kube-api-access-77r4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:56.079359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:56.079328    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdea09b-3e82-42c2-b7e9-02d6236111a2-maas-api-xks-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:56.079359 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:56.079356    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77r4l\" (UniqueName: \"kubernetes.io/projected/3fdea09b-3e82-42c2-b7e9-02d6236111a2-kube-api-access-77r4l\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:43:56.769384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:56.769350    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks"
Oct 30 03:43:56.769384 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:56.769372    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a9c96-g5lks" event={"ID":"3fdea09b-3e82-42c2-b7e9-02d6236111a2","Type":"ContainerDied","Data":"58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1"}
Oct 30 03:43:56.769610 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:43:56.769422    4214 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c38c4501b3dc4f356942259bef1b28ef1119fd1a3fa44b2654f54ddd5cd2f1"
Oct 30 03:44:04.820125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.820092    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:04.820765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.820540    4214 cpu_manager.go:414] "containerMap: removing container" podUID="3fdea09b-3e82-42c2-b7e9-02d6236111a2" containerName="revoke-keys"
Oct 30 03:44:04.820765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.820555    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdea09b-3e82-42c2-b7e9-02d6236111a2" containerName="revoke-keys"
Oct 30 03:44:04.820765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.820666    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdea09b-3e82-42c2-b7e9-02d6236111a2" containerName="revoke-keys"
Oct 30 03:44:04.824175 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.824154    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.861002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.860879    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bzc5m\""
Oct 30 03:44:04.865032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.860592    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 03:44:04.865032 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.862298    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 03:44:04.865579 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.865550    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:04.866396 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.866368    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.866518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.866427    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.866518 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.866516    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.866649 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.866541    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxg8\" (UniqueName: \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.967456 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.967421    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.967654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.967486    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.967654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.967592    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.967654 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.967628    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxg8\" (UniqueName: \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.968207 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.968180    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.969973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.969950    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.970089 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.970068    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:04.988119 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:04.988093    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxg8\" (UniqueName: \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\") pod \"maas-api-5c48877c75-q8glg\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:05.140482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:05.140410    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:05.442542 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:05.442512    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:05.443849 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:44:05.443817    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326edf37_aea6_4c79_adc4_07fd980fe363.slice/crio-5dc5f06b0f6fdc37880a3b66873e024f09516e6c8d06088c773f9eb0827c03b8 WatchSource:0}: Error finding container 5dc5f06b0f6fdc37880a3b66873e024f09516e6c8d06088c773f9eb0827c03b8: Status 404 returned error can't find the container with id 5dc5f06b0f6fdc37880a3b66873e024f09516e6c8d06088c773f9eb0827c03b8
Oct 30 03:44:05.445389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:05.445367    4214 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:44:05.821225 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:05.821192    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" event={"ID":"326edf37-aea6-4c79-adc4-07fd980fe363","Type":"ContainerStarted","Data":"5dc5f06b0f6fdc37880a3b66873e024f09516e6c8d06088c773f9eb0827c03b8"}
Oct 30 03:44:06.828108 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:06.828070    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" event={"ID":"326edf37-aea6-4c79-adc4-07fd980fe363","Type":"ContainerStarted","Data":"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"}
Oct 30 03:44:06.828552 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:06.828146    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:06.875342 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:06.875292    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" podStartSLOduration=2.449792016 podStartE2EDuration="2.875277225s" totalImagesPullingTime="425.485209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:06.873480064 +0000 UTC m=+3785.947650858" watchObservedRunningTime="2026-10-30 03:44:06.875277225 +0000 UTC m=+3785.949448019"
Oct 30 03:44:08.014906 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:08.014875    4214 scope.go:122] "RemoveContainer" containerID="7398d690a165d7dbc495c9de4c44e2fb3b842519ee6403c5ac65fde0d55e4fab"
Oct 30 03:44:08.024062 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:08.024042    4214 scope.go:122] "RemoveContainer" containerID="b65e37e87a383a6bd4b5c557b9bbd8896e06c502a44ea94c34e465f0889f5ab5"
Oct 30 03:44:09.164272 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:44:09.164211    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:44:12.046844 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.046802    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"]
Oct 30 03:44:12.059457 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.059421    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6nbdj"]
Oct 30 03:44:12.501834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.501741    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:12.502010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.501983    4214 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" podUID="326edf37-aea6-4c79-adc4-07fd980fe363" containerName="maas-api" containerID="cri-o://1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42" gracePeriod=30
Oct 30 03:44:12.506871 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.506849    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:12.563301 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.563211    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"]
Oct 30 03:44:12.567026 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.567008    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.587975 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.587949    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"]
Oct 30 03:44:12.648306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.648271    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-metrics-certs\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.648306 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.648320    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-maas-api-tls\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.648550 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.648407    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d49l\" (UniqueName: \"kubernetes.io/projected/a9053867-fd88-4b65-b071-d51191ce3dd5-kube-api-access-9d49l\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.648604 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.648568    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9053867-fd88-4b65-b071-d51191ce3dd5-openshift-service-ca\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.750066 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.750032    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9053867-fd88-4b65-b071-d51191ce3dd5-openshift-service-ca\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.750206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.750156    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-metrics-certs\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.750206 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.750184    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-maas-api-tls\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.750366 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.750215    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d49l\" (UniqueName: \"kubernetes.io/projected/a9053867-fd88-4b65-b071-d51191ce3dd5-kube-api-access-9d49l\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.750822 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.750790    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9053867-fd88-4b65-b071-d51191ce3dd5-openshift-service-ca\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.752609 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.752564    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-metrics-certs\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.752695 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.752649    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9053867-fd88-4b65-b071-d51191ce3dd5-maas-api-tls\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.772824 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.772804    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:12.772937 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.772802    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d49l\" (UniqueName: \"kubernetes.io/projected/a9053867-fd88-4b65-b071-d51191ce3dd5-kube-api-access-9d49l\") pod \"maas-api-595cc648c-zd8gm\" (UID: \"a9053867-fd88-4b65-b071-d51191ce3dd5\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.851634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.851598    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\") pod \"326edf37-aea6-4c79-adc4-07fd980fe363\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") "
Oct 30 03:44:12.851797 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.851771    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\" (UniqueName: \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\") pod \"326edf37-aea6-4c79-adc4-07fd980fe363\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") "
Oct 30 03:44:12.851842 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.851820    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\") pod \"326edf37-aea6-4c79-adc4-07fd980fe363\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") "
Oct 30 03:44:12.851919 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.851889    4214 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\") pod \"326edf37-aea6-4c79-adc4-07fd980fe363\" (UID: \"326edf37-aea6-4c79-adc4-07fd980fe363\") "
Oct 30 03:44:12.852339 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.852314    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca" pod "326edf37-aea6-4c79-adc4-07fd980fe363" (UID: "326edf37-aea6-4c79-adc4-07fd980fe363"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:12.854182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.854156    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8" pod "326edf37-aea6-4c79-adc4-07fd980fe363" (UID: "326edf37-aea6-4c79-adc4-07fd980fe363"). InnerVolumeSpecName "kube-api-access-cpxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:12.854355 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.854333    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs" pod "326edf37-aea6-4c79-adc4-07fd980fe363" (UID: "326edf37-aea6-4c79-adc4-07fd980fe363"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:12.854423 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.854356    4214 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls" pod "326edf37-aea6-4c79-adc4-07fd980fe363" (UID: "326edf37-aea6-4c79-adc4-07fd980fe363"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:12.864222 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.864195    4214 generic.go:356] "Generic (PLEG): container finished" podID="326edf37-aea6-4c79-adc4-07fd980fe363" containerID="1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42" exitCode=0
Oct 30 03:44:12.864332 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.864278    4214 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"
Oct 30 03:44:12.864380 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.864276    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" event={"ID":"326edf37-aea6-4c79-adc4-07fd980fe363","Type":"ContainerDied","Data":"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"}
Oct 30 03:44:12.864419 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.864389    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-q8glg" event={"ID":"326edf37-aea6-4c79-adc4-07fd980fe363","Type":"ContainerDied","Data":"5dc5f06b0f6fdc37880a3b66873e024f09516e6c8d06088c773f9eb0827c03b8"}
Oct 30 03:44:12.864458 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.864417    4214 scope.go:122] "RemoveContainer" containerID="1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"
Oct 30 03:44:12.875142 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.875125    4214 scope.go:122] "RemoveContainer" containerID="1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"
Oct 30 03:44:12.875495 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:44:12.875475    4214 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42\": container with ID starting with 1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42 not found: ID does not exist" containerID="1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"
Oct 30 03:44:12.875584 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.875517    4214 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42"} err="failed to get container status \"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42\": rpc error: code = NotFound desc = could not find container \"1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42\": container with ID starting with 1f8c2a6068cef7faf84487ffd63cd381a066176677bff98cac0556a779bd0c42 not found: ID does not exist"
Oct 30 03:44:12.903956 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.903927    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:12.919409 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.919189    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:12.925504 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.925476    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-q8glg"]
Oct 30 03:44:12.953887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.953839    4214 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-maas-api-tls\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:44:12.953887 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.953879    4214 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxg8\" (UniqueName: \"kubernetes.io/projected/326edf37-aea6-4c79-adc4-07fd980fe363-kube-api-access-cpxg8\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:44:12.954140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.953895    4214 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326edf37-aea6-4c79-adc4-07fd980fe363-metrics-certs\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:44:12.954140 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:12.953911    4214 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326edf37-aea6-4c79-adc4-07fd980fe363-openshift-service-ca\") on node \"ip-10-0-136-171.ec2.internal\" DevicePath \"\""
Oct 30 03:44:13.066160 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.066123    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326edf37-aea6-4c79-adc4-07fd980fe363" path="/var/lib/kubelet/pods/326edf37-aea6-4c79-adc4-07fd980fe363/volumes"
Oct 30 03:44:13.066673 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.066618    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cfd86c-c5a8-458c-917d-d329b6233fab" path="/var/lib/kubelet/pods/63cfd86c-c5a8-458c-917d-d329b6233fab/volumes"
Oct 30 03:44:13.213003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.212977    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"]
Oct 30 03:44:13.215183 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:44:13.215156    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9053867_fd88_4b65_b071_d51191ce3dd5.slice/crio-dac8c98429ceb2da8357123101a52c46327f563a0da870a9c55b63526ba457e4 WatchSource:0}: Error finding container dac8c98429ceb2da8357123101a52c46327f563a0da870a9c55b63526ba457e4: Status 404 returned error can't find the container with id dac8c98429ceb2da8357123101a52c46327f563a0da870a9c55b63526ba457e4
Oct 30 03:44:13.873878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.873776    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm" event={"ID":"a9053867-fd88-4b65-b071-d51191ce3dd5","Type":"ContainerStarted","Data":"c4d6c84893a99aedd49f15b39568930fe55165884784d6e83519b1715a4f82f0"}
Oct 30 03:44:13.873878 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.873843    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm" event={"ID":"a9053867-fd88-4b65-b071-d51191ce3dd5","Type":"ContainerStarted","Data":"dac8c98429ceb2da8357123101a52c46327f563a0da870a9c55b63526ba457e4"}
Oct 30 03:44:13.874090 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.873958    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:13.920688 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:13.920621    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm" podStartSLOduration=1.599930277 podStartE2EDuration="1.920605391s" totalImagesPullingTime="320.675114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:13.917841083 +0000 UTC m=+3792.992011876" watchObservedRunningTime="2026-10-30 03:44:13.920605391 +0000 UTC m=+3792.994776178"
Oct 30 03:44:19.887276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:44:19.887223    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zd8gm"
Oct 30 03:44:39.228065 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:44:39.228028    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:45:08.050205 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:08.050169    4214 scope.go:122] "RemoveContainer" containerID="720d8bd785e1d1f47ae4c46d516b836468b1314e83d8f7171a5ab4035cf69dc1"
Oct 30 03:45:08.061410 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:08.061385    4214 scope.go:122] "RemoveContainer" containerID="79404c58ef3079efea18bedeb65e84da70691704b3415611b75c04b6bc24a646"
Oct 30 03:45:09.163795 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:45:09.163752    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:45:14.074060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:14.074030    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74685b6749-zmk6j_0088fc29-fd63-4aa4-b655-c8482f25fbdb/manager/0.log"
Oct 30 03:45:14.567973 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:14.567942    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4zg7h_bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9/manager/0.log"
Oct 30 03:45:15.064536 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:15.064497    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:45:16.434836 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:16.434798    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b8rpz_ae0dbaef-117a-4285-aa15-47af049ee41e/manager/0.log"
Oct 30 03:45:16.462940 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:16.462914    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b8rpz_ae0dbaef-117a-4285-aa15-47af049ee41e/copy-manifests/0.log"
Oct 30 03:45:16.674704 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:16.674671    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-gtpxx_c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44/proxy/0.log"
Oct 30 03:45:17.658494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:17.658454    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zd8gm_a9053867-fd88-4b65-b071-d51191ce3dd5/maas-api/0.log"
Oct 30 03:45:18.223514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:18.223478    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-pp4kk_026fc7fa-dff5-4d51-b183-f8e313889c11/revoke-keys/0.log"
Oct 30 03:45:18.421583 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:18.421554    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-p7x8q_7a3774b3-c144-4016-8262-b887353bcdbb/revoke-keys/0.log"
Oct 30 03:45:18.802672 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:18.802635    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4ghmz_c094906a-e596-4365-b96d-c370af503668/revoke-keys/0.log"
Oct 30 03:45:18.998634 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:18.998601    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-2h2dm_95de62b6-e6ab-4a1b-a3a0-1e68fef81774/revoke-keys/0.log"
Oct 30 03:45:19.192728 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:19.192655    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-efda0460-klbfv_83e70755-392b-4b77-8249-b0d7aa5d67b3/revoke-keys/0.log"
Oct 30 03:45:19.578636 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:19.578607    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f58a9c96-g5lks_3fdea09b-3e82-42c2-b7e9-02d6236111a2/revoke-keys/0.log"
Oct 30 03:45:19.782267 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:19.782189    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2lj4q_db49e973-a835-45e7-8513-9f5315159095/postgres/0.log"
Oct 30 03:45:21.722277 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:21.722212    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-d6ndt_098ffa02-d582-4f82-96bc-3767ef975589/manager/0.log"
Oct 30 03:45:21.921411 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:21.921383    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-qqpht_4288f6cc-f324-4098-a8ab-a59d418fcc2b/manager/0.log"
Oct 30 03:45:22.338717 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:22.338684    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-2xn6x_2b082597-6fd3-484a-9e6b-d9d400d0921a/manager/0.log"
Oct 30 03:45:23.819125 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:23.819083    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t7cnx_73fccb64-3ddb-4f4b-860c-b4953d677916/discovery/0.log"
Oct 30 03:45:24.052804 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:24.052750    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-j22s7_e261b600-8177-4830-9501-e26f627e6b8f/kube-auth-proxy/0.log"
Oct 30 03:45:25.724592 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:25.724556    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-qv642_583b27ad-575a-4c4c-b3aa-9f512e025a17/istio-proxy/0.log"
Oct 30 03:45:25.914407 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:25.914352    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rl5mn_0a13416b-810d-4f7c-9aa8-e3809e65ccfa/istio-proxy/0.log"
Oct 30 03:45:26.330487 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:26.330458    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xmv89_f4e682e6-c3a8-434f-9fae-06137c443148/istio-proxy/0.log"
Oct 30 03:45:26.522403 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:26.522329    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8tfqx_de64f1d2-dc08-4bc0-9c69-f7b588b88c46/payload-pre-processing/0.log"
Oct 30 03:45:27.727482 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:27.727431    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt_74cb6b35-cc28-4b86-be20-3d994dd660bc/main/0.log"
Oct 30 03:45:27.917002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:27.916971    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fmtkt_04056afb-ee76-4d83-a900-95fc26f11e7c/main/0.log"
Oct 30 03:45:28.106564 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:28.106538    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-cc52z_2af9b4df-18a9-404e-b4f2-e1df80947697/main/0.log"
Oct 30 03:45:28.490651 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:28.490578    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4_63d54bdc-70b6-4c6c-8ab2-4dcab5038bab/main/0.log"
Oct 30 03:45:38.809244 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:38.809179    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-74685b6749-zmk6j_0088fc29-fd63-4aa4-b655-c8482f25fbdb/manager/0.log"
Oct 30 03:45:39.164575 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:45:39.164487    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:45:39.233763 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:39.233726    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4zg7h_bc2cad7e-ef2a-4091-b6bf-d3ceab7557f9/manager/0.log"
Oct 30 03:45:39.660028 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:39.659989    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:45:41.068611 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:41.068578    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b8rpz_ae0dbaef-117a-4285-aa15-47af049ee41e/copy-manifests/0.log"
Oct 30 03:45:41.099239 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:41.099199    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-b8rpz_ae0dbaef-117a-4285-aa15-47af049ee41e/manager/0.log"
Oct 30 03:45:41.292507 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:41.292478    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-gtpxx_c1cb0eed-1e68-4dbf-b45b-6de4c48d9f44/proxy/0.log"
Oct 30 03:45:42.034600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:42.034548    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zd8gm_a9053867-fd88-4b65-b071-d51191ce3dd5/maas-api/0.log"
Oct 30 03:45:42.409653 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:42.409625    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-pp4kk_026fc7fa-dff5-4d51-b183-f8e313889c11/revoke-keys/0.log"
Oct 30 03:45:42.601700 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:42.601670    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-p7x8q_7a3774b3-c144-4016-8262-b887353bcdbb/revoke-keys/0.log"
Oct 30 03:45:42.983830 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:42.983800    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4ghmz_c094906a-e596-4365-b96d-c370af503668/revoke-keys/0.log"
Oct 30 03:45:43.180530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:43.180493    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-2h2dm_95de62b6-e6ab-4a1b-a3a0-1e68fef81774/revoke-keys/0.log"
Oct 30 03:45:43.372182 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:43.372154    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-efda0460-klbfv_83e70755-392b-4b77-8249-b0d7aa5d67b3/revoke-keys/0.log"
Oct 30 03:45:43.762003 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:43.761926    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f58a9c96-g5lks_3fdea09b-3e82-42c2-b7e9-02d6236111a2/revoke-keys/0.log"
Oct 30 03:45:43.940526 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:43.940492    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2lj4q_db49e973-a835-45e7-8513-9f5315159095/postgres/0.log"
Oct 30 03:45:45.296280 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:45.296212    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-d6ndt_098ffa02-d582-4f82-96bc-3767ef975589/manager/0.log"
Oct 30 03:45:45.492473 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:45.492442    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-qqpht_4288f6cc-f324-4098-a8ab-a59d418fcc2b/manager/0.log"
Oct 30 03:45:45.871425 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:45.871392    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-2xn6x_2b082597-6fd3-484a-9e6b-d9d400d0921a/manager/0.log"
Oct 30 03:45:47.086120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:47.086087    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t7cnx_73fccb64-3ddb-4f4b-860c-b4953d677916/discovery/0.log"
Oct 30 03:45:47.293013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:47.292987    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-j22s7_e261b600-8177-4830-9501-e26f627e6b8f/kube-auth-proxy/0.log"
Oct 30 03:45:48.819829 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:48.819792    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-qv642_583b27ad-575a-4c4c-b3aa-9f512e025a17/istio-proxy/0.log"
Oct 30 03:45:49.019401 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:49.019367    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rl5mn_0a13416b-810d-4f7c-9aa8-e3809e65ccfa/istio-proxy/0.log"
Oct 30 03:45:49.423946 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:49.423881    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xmv89_f4e682e6-c3a8-434f-9fae-06137c443148/istio-proxy/0.log"
Oct 30 03:45:49.612485 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:49.612456    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8tfqx_de64f1d2-dc08-4bc0-9c69-f7b588b88c46/payload-pre-processing/0.log"
Oct 30 03:45:50.523534 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:50.523494    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hmfmt_74cb6b35-cc28-4b86-be20-3d994dd660bc/main/0.log"
Oct 30 03:45:50.716978 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:50.716949    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fmtkt_04056afb-ee76-4d83-a900-95fc26f11e7c/main/0.log"
Oct 30 03:45:50.908471 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:50.908432    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-cc52z_2af9b4df-18a9-404e-b4f2-e1df80947697/main/0.log"
Oct 30 03:45:51.305514 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:51.305476    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b894wx4_63d54bdc-70b6-4c6c-8ab2-4dcab5038bab/main/0.log"
Oct 30 03:45:57.027389 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:57.027354    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"]
Oct 30 03:45:57.039765 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:57.039736    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2h2dm"]
Oct 30 03:45:57.065033 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:45:57.065003    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95de62b6-e6ab-4a1b-a3a0-1e68fef81774" path="/var/lib/kubelet/pods/95de62b6-e6ab-4a1b-a3a0-1e68fef81774/volumes"
Oct 30 03:46:08.091156 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:08.091118    4214 scope.go:122] "RemoveContainer" containerID="7151b5c3cc6cbfbe6e01b7cb39bd211d0590b9cf590622581d541bc814b5405a"
Oct 30 03:46:09.163616 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:46:09.163575    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:46:14.035320 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:14.035221    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"]
Oct 30 03:46:14.047729 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:14.047699    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4ghmz"]
Oct 30 03:46:15.066613 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:15.066577    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c094906a-e596-4365-b96d-c370af503668" path="/var/lib/kubelet/pods/c094906a-e596-4365-b96d-c370af503668/volumes"
Oct 30 03:46:21.704386 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:21.704350    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:46:21.976709 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:46:21.976622    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:46:39.163372 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:46:39.163332    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:47:02.220908 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:02.220872    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:47:02.497009 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:02.496904    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6658f68bfc-lz8dz_467b1e74-a20c-4551-97a1-899a1e1f6ae0/manager/0.log"
Oct 30 03:47:06.027034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:06.026996    4214 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"]
Oct 30 03:47:06.039263 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:06.039219    4214 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efda0460-klbfv"]
Oct 30 03:47:07.065120 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:07.065084    4214 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e70755-392b-4b77-8249-b0d7aa5d67b3" path="/var/lib/kubelet/pods/83e70755-392b-4b77-8249-b0d7aa5d67b3/volumes"
Oct 30 03:47:08.112944 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:08.112910    4214 scope.go:122] "RemoveContainer" containerID="d34917a4c203fedb9c760176a3f077a6db7aa1477d05488e536e6bc8deb8265c"
Oct 30 03:47:09.163727 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:47:09.163690    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"
Oct 30 03:47:10.452760 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:10.452726    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d6zml_81e5958e-50f9-4bde-948e-841d12e07b7d/global-pull-secret-syncer/0.log"
Oct 30 03:47:11.451034 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:11.451000    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5v56d_f669d42f-1e2c-43c2-8172-202d70c39f0d/konnectivity-agent/0.log"
Oct 30 03:47:11.790621 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:11.790542    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-171.ec2.internal_05848580451918f708548c7de17eb55a/haproxy/0.log"
Oct 30 03:47:18.743769 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:18.743736    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-d6ndt_098ffa02-d582-4f82-96bc-3767ef975589/manager/0.log"
Oct 30 03:47:18.865494 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:18.865469    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-qqpht_4288f6cc-f324-4098-a8ab-a59d418fcc2b/manager/0.log"
Oct 30 03:47:19.138530 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:19.138498    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-2xn6x_2b082597-6fd3-484a-9e6b-d9d400d0921a/manager/0.log"
Oct 30 03:47:20.837391 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:20.837346    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/alertmanager/0.log"
Oct 30 03:47:20.952655 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:20.952624    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/config-reloader/0.log"
Oct 30 03:47:21.008010 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.007979    4214 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"]
Oct 30 03:47:21.008600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.008580    4214 cpu_manager.go:414] "containerMap: removing container" podUID="326edf37-aea6-4c79-adc4-07fd980fe363" containerName="maas-api"
Oct 30 03:47:21.008600 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.008599    4214 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326edf37-aea6-4c79-adc4-07fd980fe363" containerName="maas-api"
Oct 30 03:47:21.008793 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.008723    4214 memory_manager.go:365] "RemoveStaleState removing state" podUID="326edf37-aea6-4c79-adc4-07fd980fe363" containerName="maas-api"
Oct 30 03:47:21.012432 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.012411    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.028780 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.028756    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"]
Oct 30 03:47:21.034007 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.033986    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkbr\" (UniqueName: \"kubernetes.io/projected/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-kube-api-access-ztkbr\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.034111 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.034047    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-sys\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.034197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.034169    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-lib-modules\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.034276 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.034260    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-proc\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.034377 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.034362    4214 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-podres\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.067008 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.066987    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/kube-rbac-proxy-web/0.log"
Oct 30 03:47:21.117382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.117327    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"openshift-service-ca.crt\""
Oct 30 03:47:21.117493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.117454    4214 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zst6n\"/\"default-dockercfg-jx5vb\""
Oct 30 03:47:21.117493 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.117480    4214 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"kube-root-ca.crt\""
Oct 30 03:47:21.135574 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135542    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-sys\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135595    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-lib-modules\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135628    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-sys\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135691 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135638    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-proc\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135692    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-proc\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135722    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-podres\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135765    4214 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkbr\" (UniqueName: \"kubernetes.io/projected/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-kube-api-access-ztkbr\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.135834 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135779    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-lib-modules\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.136001 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.135863    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-podres\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.156571 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.156550    4214 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkbr\" (UniqueName: \"kubernetes.io/projected/b4b62cd8-d1a8-4fd6-9a91-056485f4003d-kube-api-access-ztkbr\") pod \"perf-node-gather-daemonset-6gwwz\" (UID: \"b4b62cd8-d1a8-4fd6-9a91-056485f4003d\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.179892 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.179873    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/kube-rbac-proxy/0.log"
Oct 30 03:47:21.293998 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.293966    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/kube-rbac-proxy-metric/0.log"
Oct 30 03:47:21.329614 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.329587    4214 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:21.408677 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.408647    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/prom-label-proxy/0.log"
Oct 30 03:47:21.522828 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.522804    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9cab9a6-c144-4af0-94fb-ac7ecb1f396e/init-config-reloader/0.log"
Oct 30 03:47:21.630932 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:21.630883    4214 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"]
Oct 30 03:47:21.632944 ip-10-0-136-171 kubenswrapper[4214]: W1030 03:47:21.632919    4214 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4b62cd8_d1a8_4fd6_9a91_056485f4003d.slice/crio-2c7971a0e31232786d43b91a21e85b9407f0a31f474d79e0accdd127966893be WatchSource:0}: Error finding container 2c7971a0e31232786d43b91a21e85b9407f0a31f474d79e0accdd127966893be: Status 404 returned error can't find the container with id 2c7971a0e31232786d43b91a21e85b9407f0a31f474d79e0accdd127966893be
Oct 30 03:47:22.018916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.018820    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz" event={"ID":"b4b62cd8-d1a8-4fd6-9a91-056485f4003d","Type":"ContainerStarted","Data":"fc83bde78ab033c4b3abdf97db05651e7349726c5a0b923e165e9c512b4b6518"}
Oct 30 03:47:22.018916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.018885    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:22.018916 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.018906    4214 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz" event={"ID":"b4b62cd8-d1a8-4fd6-9a91-056485f4003d","Type":"ContainerStarted","Data":"2c7971a0e31232786d43b91a21e85b9407f0a31f474d79e0accdd127966893be"}
Oct 30 03:47:22.059427 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.059386    4214 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz" podStartSLOduration=2.059373361 podStartE2EDuration="2.059373361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:22.058079877 +0000 UTC m=+3981.132250670" watchObservedRunningTime="2026-10-30 03:47:22.059373361 +0000 UTC m=+3981.133544153"
Oct 30 03:47:22.098013 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.097989    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f76f89f7f-vw69h_8e2a7555-f02f-4c87-8a50-6e8699c1815b/metrics-server/0.log"
Oct 30 03:47:22.214349 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.214317    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-779ms_cfaf4064-d29b-44e0-a681-2ea5e15cf178/monitoring-plugin/0.log"
Oct 30 03:47:22.672513 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.672481    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dnxm_20219a22-8edd-4c72-97dc-a2a35a1210e8/node-exporter/0.log"
Oct 30 03:47:22.786344 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.786303    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dnxm_20219a22-8edd-4c72-97dc-a2a35a1210e8/kube-rbac-proxy/0.log"
Oct 30 03:47:22.898115 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:22.898090    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5dnxm_20219a22-8edd-4c72-97dc-a2a35a1210e8/init-textfile/0.log"
Oct 30 03:47:24.483019 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:24.482988    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tdffn_d6626491-0f7e-456e-b1ff-e8f93e2325cd/prometheus-operator/0.log"
Oct 30 03:47:24.591382 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:24.591354    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tdffn_d6626491-0f7e-456e-b1ff-e8f93e2325cd/kube-rbac-proxy/0.log"
Oct 30 03:47:28.035243 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:28.035205    4214 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-6gwwz"
Oct 30 03:47:32.553701 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:32.553675    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khc7g_701387a9-97f7-42f7-b9f8-11a044f5a3c7/dns/0.log"
Oct 30 03:47:32.662800 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:32.662763    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khc7g_701387a9-97f7-42f7-b9f8-11a044f5a3c7/kube-rbac-proxy/0.log"
Oct 30 03:47:33.351197 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:33.351161    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cqnzm_8578d008-b589-4708-b51f-f94fd8176e44/dns-node-resolver/0.log"
Oct 30 03:47:34.640840 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:34.640804    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5677cfbdd5-87kq2_f9696533-3759-4de0-9d3c-5df1d9416911/registry/0.log"
Oct 30 03:47:34.976727 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:34.976651    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtkw9_0a968709-cf63-4ebc-a35e-aff8293d815c/node-ca/0.log"
Oct 30 03:47:35.628858 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:35.628824    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t7cnx_73fccb64-3ddb-4f4b-860c-b4953d677916/discovery/0.log"
Oct 30 03:47:35.658761 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:35.658733    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-j22s7_e261b600-8177-4830-9501-e26f627e6b8f/kube-auth-proxy/0.log"
Oct 30 03:47:36.144269 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:36.144222    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-qv642_583b27ad-575a-4c4c-b3aa-9f512e025a17/istio-proxy/0.log"
Oct 30 03:47:36.203002 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:36.202977    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rl5mn_0a13416b-810d-4f7c-9aa8-e3809e65ccfa/istio-proxy/0.log"
Oct 30 03:47:36.397068 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:36.396988    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xmv89_f4e682e6-c3a8-434f-9fae-06137c443148/istio-proxy/0.log"
Oct 30 03:47:36.451805 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:36.451778    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8tfqx_de64f1d2-dc08-4bc0-9c69-f7b588b88c46/payload-pre-processing/0.log"
Oct 30 03:47:38.584060 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:38.584029    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t7cnx_73fccb64-3ddb-4f4b-860c-b4953d677916/discovery/0.log"
Oct 30 03:47:39.152890 ip-10-0-136-171 kubenswrapper[4214]: I1030 03:47:39.152846    4214 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-j22s7_e261b600-8177-4830-9501-e26f627e6b8f/kube-auth-proxy/0.log"
Oct 30 03:47:39.164967 ip-10-0-136-171 kubenswrapper[4214]: E1030 03:47:39.164932    4214 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c4grk" containerName="extractor"